Gemini is getting better at understanding what’s on their phone screen

Google is updating Gemini on Android to let your AI make better use of what’s on screen. The update should allow Gemini to lean into one of its best use cases, helping you make sense of a limited set of data throughout the day.

If you set Gemini as the default assistant on your Android phone, it can already summarize or answer questions about a web page or screenshot. Soon, it will also be able to tell if there’s a video on your screen and prompt you to ask questions about it. Gemini uses the video’s automatic captions to find answers – something you could already do in a more indirect way.

Gemini will also follow a similar suggestion if you’re looking at a PDF, but there’s a catch: you’ll need access to Google’s paid version, Gemini Advanced, to use it. This is because the feature ingests the entire PDF, so it requires the long context window available to Gemini Advanced subscribers. But once you include the PDF, you’ve basically turned him into an expert on any subject—maybe it’s your dishwasher owner’s manual or your local curbside recycling guidelines. Gemini Advanced is part of the $20 per month Google One AI Premium Plan.

There’s one more small update too – you’ll soon be able to drag and drop Gemini-generated images into whatever you’re working on without needing to switch between apps. Just long-press an image in the Gemini overlay and drag it into a chat or email. All in all, it has the net effect of making Gemini feel less like something you have to pick up and something that is seamlessly integrated into the rest of the system.

It’s also a reminder that Google has been chasing this dream of context-aware search for over a decade – remember Google Now? I think it’s a remarkable advance; The best use of Gemini Assistant I’ve found is to ask it to remember a dinner recipe so I can ask questions while I’m cooking in the kitchen. It sounds simple, but it seems to me much more useful in practice than questioning the AI ​​about all the knowledge on the entire Internet.

Google Gemini updates on Android will be rolling out to “hundreds of millions of devices in the coming months” and more contextual features are in development.