AI masters Minecraft: DeepMind program finds diamonds without being taught

The Dreamer system reached the milestone by ‘imagining’ the future impact of possible decisions

*Minecraft* is among the world’s most popular video games and has 100 million monthly active users.Credit: Mojang Studios

An artificial intelligence (AI) system has for the first time figured out how to collect diamonds in the hugely popular video game Minecraft — a difficult task requiring multiple steps — without being shown how to play. Its creators say the system, called Dreamer, is a step towards machines that can generalize knowledge learned in one domain to new situations, a major goal of AI.

“Dreamer marks a significant step towards general AI systems,” says Danijar Hafner, a computer scientist at Google DeepMind in San Francisco, California. “It allows AI to understand its physical environment and also to self-improve over time, without a human having to tell it exactly what to do.” Hafner and his colleagues describe Dreamer in a study in Nature published on 2 April¹.

In Minecraft, players explore a virtual 3D world containing a variety of terrains, including forests, mountains, deserts and swamps. Players use the world’s resources to create objects, such as chests, fences and swords — and collect items, among the most prized of which are diamonds.

Importantly, says Hafner, no two experiences are the same. “Every time you play Minecraft, it’s a new, randomly generated world,” he says. This makes it useful for challenging an AI system that researchers want to be able to generalize from one situation to the next. “You have to really understand what's in front of you; you can’t just memorize a specific strategy,” he says.

Diamonds are among *Minecraft*’s most prized rewards — but finding them takes a series of complex steps.Credit: Mojang Studios

Collecting a diamond is “a very hard task”, says computer scientist Jeff Clune at the University of British Columbia in Vancouver, Canada, who was part of a separate team that trained a program to find diamonds using videos of human play². “There is no question this represents a major step forward for the field.”

Diamonds are forever

AI researchers have focused on finding diamonds, says Hafner, because it requires a series of complicated steps, including finding trees and breaking them down to gather wood, which players can use to build a crafting table.

This, together with more wood, can be used to make a wooden pickaxe — and so on, until players have assembled the correct tools to collect a diamond, which is buried deep underground. “There’s a long chain of these milestones, and so, it requires very deep exploration,” he says.

Previous attempts to get AI systems to collect diamonds relied on using videos of human play² or researchers leading systems through the steps.

By contrast, Dreamer explores everything about the game on its own, using a trial-and-error technique called reinforcement learning — it identifies actions that are likely to beget rewards, repeats them and discards others. Reinforcement learning underpins some major advances in AI. But previous programs were specialists — they could not apply knowledge in new domains from scratch.

DeepMind’s Dreamer AI played repeated runs in *Minecraft* to learn how to collect diamonds.Courtesy of Danijar Hafner

Build me a world model

Key to Dreamer’s success, says Hafner, is that it builds a model of its surroundings and uses this ‘world model’ to ‘imagine’ future scenarios and guide decision-making. Rather like our own abstract thoughts, the world model is not an exact replica of its surroundings. But it allows the Dreamer agent to try things out and predict the potential rewards of different actions using less computation than would be needed to complete those actions in Minecraft. “The world model really equips the AI system with the ability to imagine the future,” says Hafner.

This ability could also help to create robots that can learn to interact in the real world — where the costs of trial and error are much higher than in a video game, says Hafner.

Testing Dreamer on the diamond challenge was an afterthought. “We built this whole algorithm without that in mind,” says Hafner. But it occurred to the team that it was the ideal way to test whether its algorithm could work, out of the box, on an unfamiliar task.