With recent advances in generative AI, there has been a lot of discussion about how it will be used in games, virtual worlds and the metaverse. So far, most of the focus has been on how we can speed up production pipelines, given how capital and labor-intensive many parts of game development are.

However, the opportunity for generative AI extends well beyond the production process. Indeed, generative AI is already finding its way into the game loop within products like AI Dungeon, Hidden Door and Replika; likewise, AI opponents are succeeding within games like Diplomacy that requires players to understand language and develop complex strategies. These examples are only a beginning.

It got me thinking: is there a way we could establish some levels within game development to aspire to?

A few years back, the Society of Automative Engineers (SAE) developed a 5-level schema to frame what AI within vehicles would look like. Level 1 is just basic driver assistance systems like automatic braking, up through Level 5 when a vehicle would operate at human (or superhuman) performance levels, without human supervision, in any circumstances.

Game developers often like “number goes up” hierarchies — so it inspired me to think about how to apply this thinking towards game development (as well as its natural extensions like simulations, metaverse and virtual worlds of any kind).

While autonomous cars will do work for us — in virtual worlds, we can express ourselves creatively and affect the product itself.

While much of the initial focus on generative AI in games has been the potential to accelerate production pipelines at game studios — a more holistic view of games must account for all of the creators who touch games.

Where creativity happens has a lot to do with where the creativity happens: outside the game or inside the game — and inside an established game system, or built from the ground up. Here are the main participants in the creative continuum:

• The game studios (which can include everything from a one-person indie all the way up to a massive AAA team): these developers create the experience, using engines & tools that exist outside a particular game framework. Examples include games built on Unity and Unreal or bespoke engines; Elden Ring, Grand Theft Auto, Call of Duty, World of Warcraft and most games that have ever been made.
• The game itself can create content while you play. This goes all the way back to roguelike games (including Nethack and its graphical successors like Diablo). Recent examples that include actual AI include AI Dungeon; Replika; or the products Hidden Door is talking about.
• The player is given opportunities for creative expression within the game loop. To some extent, this is every game that exists — because there’s almost always creativity in the form of tactics and strategy. Games that exist to provide a form of “sandbox” play such as Minecraft, Sim City, Star Wars Galaxies make creative expression one of the main reasons to play. There’s also the social creativity that exists through guilds and various cooperative and competitive mechanics that have emerged in MMOs, as well as creativity around the meta-experience such as livestreaming.
• Modders extend the experience of the game by using tools outside of the game loop. This includes Minecraft mods, Terraris mods, World of Warcraft UI mods. Older examples date back to Neverwinter Nights and Quake. Before even those, there was a hacker community who used sector editors and disassemblers to make their own versions of some of the earliest games.

Today, most commercial games are made by game studios (although there are plenty of notable exceptions within platforms like Roblox). Here, the levels primarily pertain to how quickly you can get ideas out of your head and onto the screen:

• Level 0: No automation. No significant automation (but might kitbash a bit). Creator builds their own engine & backend.
• Level 1: Creator Assistance. Uses 3D engine and/or backend platform as compositional framework. Leverages off-the-shelf tools and code.
• Level 2: Partial Automation. Non-technical tools for creative inputs. Tech eliminates previous steps. Serverless & containers adopted in online games.
• Level 3: Conditionally Generative. Generative AI used for some parts of the pipeline. Human heavily in the loop, concept/story iteration, code gen.
• Level 4: Highly Generative. Total automation of at least one creative pipeline, e.g., text prompt to make completely usable 3D models (which would require all the complexity to be removed from things like UV unwrapping, baking lights, etc.)
• Level 5: Direct-from-imagination. The “holodeck” level of creativity. Multiple generative pipelines are integrated comprehensively into a composition.

The modder is someone who makes experiences that fit within an established game system, rather than builds their own game from the ground up.

• Level 0: Hackers. The way it all began: people with old-fashioned sector editors, disassemblers, etc.
• Level 1: Game-provided tools. The game comes with tools such as map-editors or accessible interfaces to apply off-the-shelf creative tools.
• Level 2: Scripting interfaces. Game provides an API to build mods, or provides an embedded script interface for creating mods.
• Level 3: Conditionally Generative. Generative AI used for some parts of the pipeline (concept art, story, etc.) with a human heavily in the loop.
• Level 4: Highly Generative. Integrated generative tooling, e.g., tools like map editors invoke generative AI to accelerate creation.
• Level 5: Direct-from-Imagination. Complete mod is spawned from an idea. Iterative ”sculpting” of the mod through prompts used for refinement.

Players are also creative. Today, where blending the distinction between players and modders — and someday we will start seeing games that allow players to do many of the things that require modding today.

• Level 0: Limited Creativity. Just the tactics and strategy you use to master the game.
• Level 1: Player-as-content. The player alters a game by affecting social systems & economy, but doesn’t meaningfully impact the rest of the built-in content.
• Level 2: Worldbuilding. The player impacts the shape of the world itself, such as how maps and structures live in a persistent world.
• Level 3: Emergent Narrative. The player’s actions affect persistent content of the world, like living narratives and content — such as we’ve seen in most MMORPGs, and especially Eve Online.
• Level 4: Integrated Generativity. Things formerly the domain of modding become part of the game loop, along with in-loop generative AI.
• Level 5: Direct-from-Imagination. If players can dream of it, they speak it into existence; game system fits it to rules in an appropriate manner.

Lastly, the game itself is a source of generative content. As generative AI technologies improve, we’ll increasingly see more sophisticated content, lifelike characters and game systems that we currently can’t imagine.

• Level 0: Static Games. The game doesn’t change in significant ways between gameplay experiences. Set level progression, etc.
• Level 1: Procedural Content. Rule-based procedural content such as unique maps during play. May also include basic AI opponents.
• Level 2: Live Generative Content. Experience includes content generated “live” while playing: text, images, music or NLP-aware characters.
• Level 3: Conditionally Generative. Generative content in-the-loop utilizes rule-based systems (such as RLHF) to maintain consistency & structure.
• Level 4: Highly Generative. “Game as Dungeon Master,” the game understands what a player likes most and personalizes/adapts experience.
• Level 5: Full Computational Creativity. Explores the search space of human fun: like a game-designer friend who comes up with new systems.

What level of generative technology do you aspire to in your own games and virtual worlds? What sort of features do you imagine? I encourage everyone to look beyond the productivity-enhancing tools (as astonishing and important as they are) to probe what it will mean to have generative AI as part of the core game loop of entirely new experiences.

• The Direct-from-Imagination Era Has Begun is where I present the converging technologies that will enable us to go instantly from idea to experience.
• Composability is the Most Powerful Creative Force in the Universe introduces some concepts about how we’re always building on the shoulders of giants — and how technologies like frameworks and AI will play an important part.
• Computational Creativity is my article on how an individual person will eventually make something at the level of sophstication present in Elden Ring. I originally guessed it would take a decade, but that’s starting to seem too long.

If you enjoyed this, you may also like having the core concepts in a convenient deck format to share with your colleagues and friends: