It was only a matter of time before someone combined Stable Diffusion AI image generator and Minecraft command-line. This is probably one of the best uses of AI image generators I've seen so far, essentially using a combination of Minecraft screenshots and text prompts to create highly detailed images each time the user makes changes to their design within the Minecraft game world.
Sean Simon, more commonly known as ThoseSixFaces (or Lozmosis on Reddit), posted the above video yesterday showing off a sped-up version of their scripts running live as they messed around in-game.
It works by taking screenshots from a specific angle as the game is played, resizing or cropping them to 512x512 pixels and saving them to a folder. Each frame is generated on "a fixed interval, or whenever the player places/removes a block." Using a version of InvokeAI that Sean modified, the model then goes about "targeting the latest image in the folder and outputting the Stable Diffusion image." Another script then displays the output image using tkinter.
Stable Diffusion is the main thing translating each frame into gorgeous AI generated images, however, which Sean explains "denoises on a random value between 0.47-0.5." So although Sean is able to use prompts to push the AI in the preferred direction using image2image, there's still a little leeway in how random the generated results can be.
As it stands right now, the process involves running several scripts, though Sean is "keen to compress this all into a single script."
Best CPU for gaming: The top chips from Intel and AMD
Best gaming motherboard: The right boards
Best graphics card: Your perfect pixel-pusher awaits
Best SSD for gaming: Get into the game ahead of the rest
It's a fascinating project, and one my art teachers from college would be very excited to see working; they seemed to have an obsession with translating work into different mediums, then translating it back. On that note, one Reddit commenter jokingly suggests a reverse process to Sean, in which the AI would "turn arts into minecrafts."
Sean responds with, "I've started cooking something up," so it may not be too long before we see Minecraft worlds being generated using AI image prompts. Or indeed Minecraft worlds being translated into images, then back into Minecraft worlds.
Whew, what a world we live in.
Sadly, Sean's scripts are not yet available to download from Github, but do keep an eye on their Twitter for updates.
Now, we've seen Stable Diffusion being used in VR, and the CEO of Discord image bot Midjourney speculates that future consoles will have a "giant AI chip and all the games will be dreams." This Minecraft project is yet another addition to the exciting, AI-focussed narrative springing up in the gaming sphere right now.
