Posted: 3/21/2017 2:37 PM
Post subject: Extracting frames for an external program
Quote
eatyourvegetables
Joined: 3/21/2017
Posts: 3
Hello, I'm an ai developer, and interested in trying my claws in making ai for simple games, such as Super Mario World or even simpler. I would like to extract each frame (like, an array of exact pixel values) of a game to feed it to an analysing script (to utilize computer vision methods and whatnot) and give back a list of inputs to the game. I tried reading BizHawk documentation about Lua scripting, but I didn't find anything related to reading the screen, only drawing on it. Do you have any idea how to accomplish this?
Posted: 3/21/2017 3:10 PM
Quote
Masterjun
He/Him
Site Developer, Skilled player (1987)
Joined: 10/12/2010
Posts: 1185
Location: Germany
Exotic platforms TAS of 2020 Speedy TAS of 2018 PC TAS of 2018 Glitchy TAS of 2015 TASer of 2014 SNES TASer of 2014 TAS of 2014 SNES TAS of 2014 Funny TAS of 2014 Glitchy TAS of 2014 TASer of 2013 SNES TASer of 2013 NES TAS of 2013 SNES TAS of 2012 Funny TAS of 2012
There are several ways to accomplish what you want to do. First thing is doing exactly what you wanted to do, only with lsnes, because there you can just 
Language: lua
local srcn = gui.screenshot_bitmap()
local pixelcolor = srcn:pget(152,168) -- x:152, y:168
Secondy you can stay with BizHawk, using the client.screenshot() function to save a .png and then immediately load it using some Lua library or your own functions decoding the image, reading the pixels etc. Finally you can also simply get values from RAM that tell you the position of Mario, enemies and obstacles. But this might not fit into your computer vision methods since they don't give you pixel data. So this would result in a different kind of AI.
Warning: Might glitch to credits I will finish this ACE soon as possible (or will I?)
Posted: 3/21/2017 5:11 PM
Quote
Amaraticando
It/Its
Editor, Player (159)
Joined: 1/10/2012
Posts: 673
Location: Brazil
TAS of 2015 SNES TAS of 2015
Check this: https://github.com/aleju/mario-ai This project contains code to train a model that automatically plays the first level of Super Mario World using only raw pixels as the input (no hand-engineered features). The used technique is deep Q-learning, as described in the Atari paper (Summary), combined with a Spatial Transformer. This repo is for the lsnes emulator.
Posted: 3/21/2017 8:50 PM
Quote
brunovalads
Editor, Player (175)
Joined: 4/7/2015
Posts: 331
Location: Porto Alegre, RS, Brazil
Masterjun wrote:
some Lua library or your own functions decoding the image, reading the pixels etc.
For this you can use Lua-GD, it's nice to use, has a lot of functions.
Games are basically math with a visual representation of this math, that's why I make the scripts, to re-see games as math. My things: YouTube, GitHub, Pastebin, Twitter
Posted: 3/22/2017 9:39 AM
Quote
eatyourvegetables
Joined: 3/21/2017
Posts: 3
Thank you all for your input! I didn't find a function 
gui.screenshot_bitmap()
from http://tasvideos.org/Lsnes/LuaFunctions.html#GuiScreenshot Is this the right place? I believe taking a screenshot and saving it each frame might be terribly slow, although I have a decent computer and the resolution of old consoles is not very high. Thanks for the link to Mario-ai, I'll look into it. I come from the machine learning world, and both using pixels and reading game state from memory are interesting options. How do I know where is the current frame stored in the memory?
Posted: 3/22/2017 1:39 PM
Quote
Amaraticando
It/Its
Editor, Player (159)
Joined: 1/10/2012
Posts: 673
Location: Brazil
TAS of 2015 SNES TAS of 2015
This wiki page is horribly outdated. When you download lsnes, there's a lua.pdf inside with (almost) all functions [latest commit here]. For instance, movie.currentframe() Return number of current frame.