Google says Gemini AI is making its robots smarter

Google is training its robots with Gemini AI so they can improve navigation and task completion. The DeepMind robotics team explained in a new research article How using Gemini 1.5 Pro’s long context window – which determines how much information an AI model can process – allows users to more easily interact with their R2-T robots using natural language instructions.

This works by filming a video tour of a designated area, such as a home or office, with researchers using the Gemini 1.5 Pro to have the robot “watch” the video to learn about the environment. The robot can then execute commands based on what it observes using verbal and/or image output – such as guiding users to an electrical outlet after seeing a phone and asking “where can I charge it?” DeepMind claims its Gemini-powered robot has had a 90% success rate on more than 50 user instructions delivered across an operating area of ​​more than 9,000 square feet.

The researchers also found “preliminary evidence” that Gemini 1.5 Pro allowed its droids to plan how to carry out instructions beyond navigation. For example, when a user with many cans of Coca-Cola on their desk asks the android if their favorite drink is available, the team said Gemini “knows that the robot should navigate to the refrigerator, inspect it for Coca-Cola, and then return to the refrigerator.” user to report the result. DeepMind says it plans to further investigate these results.

The video demonstrations provided by Google are impressive, although the obvious cuts after the android recognizes each request hide that it takes 10 to 30 seconds to process those instructions, according to the research paper. It may be some time before we share our homes with more advanced environmental mapping robots, but at least they will be able to find our lost keys or wallets.