At WWDC 2024, AI could make Siri the voice assistant Apple always wanted

When Apple first released Siri in 2011 alongside the iPhone 4S, the company made a series of very eye-catching ads showing how you can use this new voice assistant. In one, Zooey Deschanel asks on the phone about tomato soup delivery; in another, John Malkovich asks for some existential life advice. There’s also one with Martin Scorsese altering his schedule in the backseat of a New York taxi. They showed reminders, weather forecast, alarms and more. The point of the ads was that Siri was a helpful, constant companion who could solve whatever you needed. No apps or taps required. Just ask.

Siri was a big deal for Apple. At the 4S launch event, Apple’s Phil Schiller said that Siri was the new device’s best feature. “For decades, technologists have teased us with the dream that we would be able to talk to technology and it would do things for us,” he said. “But it never comes true!” All we really want to do, he said, is talk to our device however we want and get information and help. In a moment of classic Apple bravado, Schiller proclaimed that Apple had solved the problem.

Apple hasn’t resolved this. In the 13 years since that initial release, Siri has become, for most people, either a way to set timers or a useless feature to be avoided at all costs. Siri has been bad for a long time, long enough that for years it seemed like Apple either forgot about it or simply chose to pretend it didn’t exist.

But next week at WWDC, if the rumors and reports are true, we could be about to meet the real Siri for the first time – or at least something much closer to it. According to Bloomberg, The New York Times, and others, Apple will unveil a major overhaul for the assistant, making Siri more reliable thanks to great language models, but without many new features. Even that would be a victory. But Apple also appears to be working on, and may be almost ready to release, a version of Siri that will actually be integrated into apps, meaning the assistant can take action on your device on your behalf. In theory at least, anything you can do on your phone, Siri may soon be able to do for you.

Obviously, this has always been Siri’s vision. You can even see it in the iPhone 4S commercials: these celebrities are asking Siri for help, and Siri almost never gets the job done. He provides Deschanel with a list of restaurants that mention delivery, but does not offer to order anything or show him the menu. He tells Scorsese there’s traffic, but doesn’t redirect him – and shouldn’t he already know he’ll be late for the meeting? Siri tells Malkovich to be nice to people and read a good book, but doesn’t offer any practical help. Until now, using Siri has been like having a virtual assistant whose only job is to Google things for you. What is something! But it’s not much.

Siri’s inabilities have been even more frustrating because everything it needs to be useful is right there. right there on your phone. When I want pizza, why can’t Siri check my email for the receipt from my last order, open DoorDash, enter the same order, pay with one of the cards in my Apple Wallet, and that’s it? If I have a Scorsese-level busy day, Siri seems to be right alongside all my contacts, my Slack, my email, and everything else I need to quickly move things along on my behalf. If Siri could take control of my phone like one of those remote access tools that lets someone else move your computer’s cursor, it would be unstoppable.

In fact, there are two reasons why Siri never reached its potential in this way. The first is simple: the underlying technology wasn’t good enough. If you’ve ever used Siri, you know how often it listens for names, misinterprets commands, and resorts to “here are some things I found on the web” when all you wanted was to play a podcast. This is where big language models are unequivocally very interesting, because we’ve seen how much better speech-to-text tools like Whisper are and how these models can understand language more broadly. They’re not perfect, but they’re a huge improvement over what we had before — which is why Amazon is also migrating Alexa to LLMs and Google Assistant is being hacked by Gemini.

The second reason Siri never worked is simply that neither Apple nor third-party developers figured out how it works. he must to work. How are you supposed to know what Siri can do or how to ask? How should developers integrate Siri? Even now, if you want to add a task to your to-do list app, Siri can’t just figure out which app you use. You have to say, Hey Siri, remind me to water the grass in Todoist, which is a weird phrase that makes no sense and, in my experience, fails half the time anyway. If you want to perform a multi-step action, your only option is to tinker with Shortcuts, which is a very powerful tool but doesn’t require you to write code. It’s too much for most people.

AI could also give Apple a chance to solve the entire problem. Its researchers published a paper earlier this year detailing a system called Ferret-UI, which uses an AI model to understand small details of an image on a screen. The researchers even detail how a general application using Siri might work: OpenAI’s GPT-4 does a good job of broadly understanding what an image is, and then Ferret is able to understand small regions and details. In practice, this could mean that a system says, “This is the Ticketmaster app!” and the other says, “Here’s the buy button.”

We should be skeptical about any claims Apple makes about Siri. More than a decade ago, Schiller took the stage and proclaimed that Apple had created a better voice assistant, but that didn’t happen. The same may be true now, as enthusiasm for AI continues to advance much faster than the actual technology. Humane, Rabbit, Google, and others are all working on similar ideas — “agent” is the buzzword of the summer in the AI ​​world — and no one has demonstrated they’re ready yet.

But if Apple is onto something here, this could be the first time we see the real Siri – the Siri we were promised years ago. Maybe in the next commercial, Deschanel’s tomato soup will magically appear in her house, and the Headspace app will activate to bring Malkovich some inner peace. Maybe we’ll finally get the Siri that Apple always wanted to make.