People are using Super Mario to benchmark AI now

Is the Pokémon modified into a challenging standard for AI? A team of researchers claims that Sizable Mario Bros.is even more difficult.

On Friday, Hao AI Lab, a research organization at the university of california San Diego, introduced AI into live Sizable Mario Bros.gameplay. Anthropic’s Claude 3.7 took on the challenge first, followed by Claude 3.5. Google’s Gemini 1.5 Legitimate and OpenAI’s GPT-4o faced difficulties.

It wasn’t exactly the same version of Sizable Mario Bros.as the original from 1985 to be clear. The game operated in an emulator and was integrated with a framework called GamingAgent, which allowed the AIs to control Mario.

GamingAgent, developed internally by Hao, provided overall instructions to the AI such as “If an obstacle or enemy is nearby, move/jump left to evade” along with in-game screenshots. The AI subsequently produced inputs in Python code to manage Mario’s actions.

Hao asserts that this game compelled each model to “learn” how to execute intricate maneuvers and formulate gameplay strategies. Interestingly enough, they discovered that reasoning models like OpenAI’so1,which “think through considerations step-by-step,” performed worse than “non-reasoning” models despite generally being superior on most benchmarks.

The present flashy gaming benchmarks level show what Andrej karpathy…(continue)
(…)
(…)
(…).

 
 The current flashy gaming benchmarks highlight what Andrej Karpathy…(continue).

    The current flashy gaming benchmarks highlight what Andrej Karpathy….</font>