How machine learning AI is going to change gaming forever

Over the past few days, I've watched my productivity drop (don't tell my boss) a fairly huge amount owing to a sudden explosion of consumer-friendly and highly memeable artificial intelligence (AI) machine learning (ML) tools you might have seen on social media.

Services like Craiyon that automatically generate rough AI images based on virtually any input prompt have produced a range of wild and wacky collages across the net, which you can peruse yourself over on subreddits like /r/weirddalle or Twitter accounts like @Dalle2Pics. 

Chatbots and weird AI-generated art are widely known, but what's perhaps more shocking is this more recent wave of sophistication, and how it is already making its way into the general public's consciousness. 

And hey, I admit I have a very surface-level understanding of some of this stuff, but my goal here is to try to outline and extrapolate potential applications we could see appear over the course of the Xbox Series X|S console generation. 

After spending hours generating ridiculous AI stories and pictures to bug my friends and co-workers, I began to dig a little more into how it all works, and what the implications could be for practical applications that move beyond simple memery. 

Microsoft has been making massive investments in the space. We've all heard about the Nuance purchase, which gave Microsoft control over the conversational tools that powers Apple's Siri, but Redmond has been quietly building up its own internal tools and features as it looks to what will most likely end up becoming the next major shift in technology, both for businesses and consumers. 

How deep-learning AI will change gaming forever

Obviously, machine learning techniques have broad implications for almost every sector of life, but how they will intersect across gaming has potentially some of the broadest implications for Microsoft as a business. 

One problem the video game industry generally faces right now pertains to the gap between expectations and investment. Video games are becoming increasingly complex to make, fund, and manage, as they explode in exponential complexity and graphical fidelity. We've seen absolutely insane Unreal Engine demos that showcase near-photorealistic scenes and graphics, but the manual labor involved to produce a full game based on some of these principles is truly palpable both in terms of time, and expense. 

What is typically thought of as "AI" in a gaming context generally hasn't been AI in the true sense of the word. Video game non-player characters (NPCs) and enemies generally operate on a rules-based model that often has to be manually crafted by a programmer. Machine learning models are importantly far more fluid, able to produce their own rules within parameters, and respond dynamically to new information on the fly. This applies across all things, whether it's NPC behavior or content generation. 

I asked a GPT-3 ML text model to generate a quest for World of Warcraft, including objectives, and even NPC dialogue. This is a system that isn't even designed for it, leveraging random and unfiltered information from the internet to recontextualize and repackage the information based on the request. I also asked an AI to generate an evil sentient banana, with similarly fun results. 

Taking context from the internet, the AI model was able to produce a practically infinite quest dialogue, drawing down from information about World of Warcraft on websites and wiki entries to produce scarily realistic objectives that included real place names and enemy types. With direct access to more refined data and tailor-made context by a game developer, it's not hard to extrapolate how these models could be adapted to produce vast arrays of dynamic content. DALL-E and similar models are increasingly capable of producing an array of variants from any prompt, refining them with extraordinary levels of detail. It can also produce remixes of existing images. It's easy to imagine how that could extend to variations of monsters in games, or even variations of in-game behavior and animations. 

I suspect that for the foreseeable future, handmade content will remain the core of what makes a great video game truly great, but ML systems will become a primary source of supplementary content for a variety of game genres.

Imagine the next generation of Minecraft's world seed generation tools, using machine learning to include game rules, generate unique monsters, and new terrain. They could extend and improve tile-based procedurally generated levels in games like Diablo or Hades. They could generate supplementary quests for endgame play in MMO-type games like World of Warcraft or Fallout 76. 

Machine learning algorithms will supplement the construction and animation of creatures, level assets, and characters, at a speed and scale that simply isn't possible when you factor in labor costs. These systems won't (and shouldn't) replace real-life developers and artists, but should instead supplement alacrity in time-intensive work that can increase costs, lead to delays, and create pressure within a collaborative workflow. If you can have an AI handle the animation and behavior model of a simplistic background entity like a rat or a snake that is there purely for ambiance, that could free up dev time for more nuanced or complex tasks that have a greater impact on a player's experience. 

These are just a few very basic examples of how machine learning will enhance and elevate games across the Xbox Game Studios portfolio, but Microsoft will be keen to extend these services to the broader game development community as well under its Azure PlayFab brand. 

Microsoft is already poised to be a frontrunner 

Microsoft has been quietly (and not so quietly) investing quite heavily in machine learning for some time. I'd argue that the impact hasn't been fully felt yet in its gaming, but that could be about to change. 

Microsoft announced that it had acquired exclusive rights to some OpenAI models such as GPT-3 back in 2019 as part of a $1 billion investment. Since then, Microsoft has begun integrating aspects of machine learning it into some of its developer products. Copilot on GitHub and PowerApps Ideas are two services that leverage machine learning based on OpenAI models to supplement and speed up app development and programming. 

OpenAI is developing broad and extensive research into machine learning techniques loosely based on neural pathways in the human brain. You can browse some of its work on its website. Dalle 2 is producing truly insane lifelike artistic renders from simple text prompts with staggering results

Indeed, there are obvious concerns that are worth addressing when it comes to these machine learning models. And no, we're not talking about computers becoming sentient and Xbox Live becoming Skynet. 

Machine learning tools have already quite notoriously been abused to create all sorts of content that could be considered brand unsafe at best, to harmful at worst. Machine learning could lead to more sophisticated player-hostile bot behavior, for example, which could automate and trivialize gameplay. Imagine an aimbot that not only aimed for you, but also moved and dodged bullets for you as well. 

Indeed, I was able to bypass a lot of the simple filters attached to many of the current free tools to produce quite surprisingly dark content I'm sure publishers wouldn't want to appear within a game like Minecraft, for example, which is enjoyed by millions of kids across the world. Researchers are actively exploring ways to mitigate these kinds of issues, of course. Smarter, and faster context-sensitive content filters are obviously a heavy area of investment on the part of everyone involved. 

I distinctly remember a conversation I had with someone at Microsoft the year Minecraft Earth was announced. The canceled Minecraft spin-off was supposed to let people build augmented reality creations and place them in the real world, and I recall asking how Microsoft would be able to prevent naughty creations from unscrupulous 'crafters from appearing in other players' worlds. Machine learning was discussed as an option — training some kind of AI to recognize and moderate offensive content would obviously be paramount in a game like Minecraft Earth, which was all about sharing content. Microsoft also famously used early versions of ML models to moderate its chat channels on Mixer, its similarly canceled streaming service. 

Of course, human intervention will still be needed for the foreseeable in many cases, but these examples just further represent how pervasive machine learning will be across the entire gaming horizon. Content creation, content moderation, and even community management could be supplemented by AI.

Microsoft has already begun exploring how to implement machine learning into its games. Project Paidia has already demonstrated how AI can be trained to perform collaboratively, and dynamically with human players. The page there even has a HTML5 mini-game you can test out to demonstrate the basics of how it works, and mentions how they were exploring a potential implementation in the team-based brawler Bleeding Edge at one point. Researchers at MineDojo are also building similar ML models into Minecraft, building bots that can behave more intelligently and perform player-like actions. Microsoft also baked DirectML into the Xbox Series X and Xbox Series S to provide compute to generic machine learning models. Microsoft also leverages the tech to improve the delivery of streamed games on Xbox Game Pass' cloud gaming platform. 

Microsoft is hiring engineers for its gaming AI efforts, led by MSFT Research alumna and computer scientist Haiyan Zhang, with the goal of implementing machine learning into game development and gameplay. Microsoft is also one of the few companies with cloud computers pervasive enough to support the processing overhead that could emerge as a result of implementing some of these models directly into service games — where they're naturally going to see their full potential realized. 

Indeed, I suspect it won't be long before we start seeing more tangible player-driven interactions using machine learning models at a gameplay level in the coming years ahead. And Microsoft already has the infrastructure, investments, and ingenuity in place to see it all come to fruition.

This could be as big as the leap from 2D to 3D

I don't like making predictions because whenever I put text to paper like this, it opens me up to being WRONG, and how scary would that be?!

You don't have to go far to see failed editorial tech predictions, like the Daily Mail's infamous "The Internet is a passing fad" article from 20 years ago. Facebook (now known as Meta) has staked its entire future seemingly on VR and the so-called metaverse, which has yet to bear fruit and seems like something of a hype bubble poised to pop. Zuckerberg himself even admitted that the metaverse won't be a thing for at least a decade only a couple of years ago, which makes you wonder how long investors will accommodate such an expensive gamble. 

Microsoft has famously been relatively lukewarm on consumer VR, but it has gone deep on AI, which to me seems like a far safer, and far more lucrative bet. The benefits for businesses are obvious and don't necessarily require a huge amount of imagination. From speedier programming to intelligent customer service chatbots, the potential is massive, but the implementation in gaming is what excites me the most. 

Obviously, there are some serious problems to solve in certain implementations, but the potential is tantalizing. Vast and dense worlds populated with realistic and meaningful art and creatures, supplementary side objectives that free up dev hours for hand-crafted features and writing. Intensive tasks that take weeks reduced to mere minutes with cloud-powered machine learning algorithms, increasing the development speed of AAA games, and democratizing AAA quality assets and animation for smaller businesses and independent developers via ID@Azure. 

It won't happen overnight, but there's a revolution for gaming just around the corner that could be on par with the leap from 2D to 3D.