Meta's AI achieves "human-level performance," defeats players in 'Diplomacy' game
Meta AI has developed a new game bot, called CICERO, for the strategic board game called Diplomacy. During the game, the AI agent not only achieved "human-level performance" but also defeated humans, while remaining anonymous. Diplomacy is a seven-member war game set in Europe, where players have to forge alliances to occupy important cities and crucial supply centers to win.
Why does this story matter?
To play the game, CICERO—the AI agent—participated with six other human players. Along with developing its own game strategy, the game bot also understood the motives and communicated with other players involved, all without getting identified as a bot. The AI imitated natural human language and built trust to beat the participants, setting a new benchmark in AI technology. Worrying?
First, what is the 'Diplomacy' board game?
Diplomacy is a seven-member board game. It was introduced in 1954. Each member takes one of the roles of the seven Great Powers in Europe before World War 1: Italy, Russia, Turkey, Austria, France, Germany, or England. The major difference from other board games is that the players do not get a chance one by one, instead all their moves are executed simultaneously.
Developing an AI for 'Diplomacy' was considered next to impossible
AI agents have been developed for games such as Chess and Poker, but creating one for Diplomacy was considered a "near-impossible grand challenge in AI," only until now. This was because the game bot, along with understanding other players' motives, would have to create and reform its own game strategy. It also had to use natural language to communicate and negotiate to form alliances.
Test demonstrations revealed that CICERO doubled the average score
The AI was trained on over 40,000 human games available on webdiplomacy.net. On the whole, these games contained over 12 million messages exchanged throughout the games. During CICERO's test demonstration on the same website, the bot more than doubled the average score of the human players and even ranked in the top 10% of participants who played more than one game.
The game bot had a tendency to overuse masculine pronouns
According to Meta's study, CICERO passed as a human player for 40 games of Diplomacy that involved 82 different players. The in-game messages revealed that the players didn't suspect that they were playing with an AI agent. Interestingly, researchers found that the game bot had an inclination toward masculine pronouns such as he/him/his- and used them 75% of the time.
The game bot occasionally contradicted its own plans
At some points during the game, CICERO sent messages that contained grounding errors. Occasionally, it even contradicted its plans and in certain other instances, the strategy was not up to par with that of other players. But the bot never got exposed. Researchers explained that even humans can commit such mistakes and owing to the time pressure, the human players didn't suspect anything.
There is still room for improvement
However, a challenge would arise if the negotiation period lengthens, where human players would indulge in more detailed conversations. At present, it is unclear how the AI's dialogues might affect relationships with other players in the long term. "Diplomacy provides a rich testbed to explore these connections between strategy and communication with the goal of improving coordination between humans and agents," noted the study.
Take a look at Meta AI's announcement
Communication is the most vital aspect of CICERO's success
"If an agent can't recognize that someone is likely bluffing or that another player would see a certain move as aggressive, it will quickly lose the game," said Meta AI in its blog. "Likewise, if it doesn't talk like a real person—showing empathy, building relationships, and speaking knowledgeably about the game—it won't find other players willing to work with it."
The technology can pave way for interesting applications in future
The technology behind CICERO is game-changing and has wider applications. What if it could hold a long-term conversation to teach you a new skill? Meta has open-sourced the codes and models used to develop CICERO to encourage other researchers to build on their work. This technology can be implemented to develop intelligent assistants that can collaborate with humans in the future.
