The problems with generative AI in the smart home and how Amazon and Google plan to fix them

Last week, Amazon and Google kick-started what could be the next chapter of the smart home. Their new voice assistants, Alexa Plus and Gemini for Home, have been rebuilt from the ground up on generative AI and large language models to be more conversational, understand context, and take actions. This marks the biggest shift in home control since the companies launched their original smart speakers over a decade ago.

In the years since, smart home adoption has stalled — because it’s complicated and confusing, and the value isn’t always clear. Google and Amazon are betting on this new wave of AI-powered intelligence to deliver a smarter, simpler, more capable smart home. After spending the last week speaking with folks at both companies and seeing their new hardware and software strategies, I’m hopeful. But I see at least three major hurdles: reliability, speed, and proving it’s worth paying for.

How generative AI could finally make the smart home smart

While you might think of generative AI as creating text and images, it can also be used to analyze data collected from a home to identify patterns and interpret context. This can provide the intelligence layer the smart home needs to move us from the command-and-control world we’ve been stuck in toward the promised land of ambient computing. With this upgrade, in theory, your home can respond and react proactively to situations, without you needing to spend hours fiddling with apps, setting up automations, or grappling with the exact phrasing required by a voice assistant.

“The biggest gap we’ve had in the last decade is that intelligence layer,” Google Home’s Anish Kattukaran tells me, adding that the command-centric nature of the current Google Assistant, Alexa, and Siri has been a major limitation for the smart home. “As an industry, we solved that with a lot of hard-coding, a lot of if-this, then-that statements.” But with generative AI and LLMs, the assistants can become that intelligence layer that we can more easily interact with, he says. “This is undoubtedly an inflection point.”

For Google, that shift starts this month, with the rollout of Gemini for Home. A “foundational intelligence” that is being added throughout Google’s smart home ecosystem, Gemini for Home is coming to all the company’s existing hardware and its new Google Home app. It will work best, however, on its newest cameras and doorbells, along with an upcoming smart speaker and possible new smart display.

The new AI-powered capabilities primarily focus on a smart home assistant that can understand and use natural language, as well as comprehend, generate, and summarize descriptions of events within and around your home. The features are being added slowly, as part of a new opt-in early access program in the US and several other countries.

Over at Amazon, its AI-upgraded Alexa Plus assistant has been in its own early access program since March. But just last week, Amazon announced that Alexa Plus will be ready to go “out of the box” on its newest hardware in the US.

Panos Panay, who leads Amazon’s devices and services division, tells me that Alexa Plus, combined with the new hardware, will create “magically connected experiences” that he believes will start to transform the smart home.

These may sound like empty promises for smart home users who have been let down year after year by the smart home’s failure to move beyond single-purpose devices and into a collectively intelligent space. Yet, having used Alexa Plus for a few months and sampled some of the features Gemini is bringing to Google Home, I can see the potential. In the distance.

The most significant immediate change is that both voice assistants can now understand what you mean — not just what you say. For example, you’ll no longer need to remember the name of every light bulb you installed or what you named the back door lock. Instead, phrases like “I’m going to cook dinner, turn the lights on” should turn the kitchen lights on, no matter which room you’re in.

They should also be better at managing your connected gadgets, making the smart home less complicated for everyone in your home. My husband has never created an Alexa Routine in his life, but recently, he wanted the lights to turn off every night at 10PM. He asked me if I could set it up, and I told him to just tell Alexa what he wanted. He did, and it worked. He was impressed — and not a little surprised.

Google has a similar, if more limited, experience. An Ask Home chatbot in the new Google Home app can respond to voice and text inputs to create routines and automations. I saw a demo where it was prompted with the query, “I want to feel safer,” and it suggested routines that simulate presence in the home, such as turning lights on and off, setting up notifications for doors or windows opening, and locking all the doors and turning off the lights when you leave.

Hurdles along the road to a smarter home

This is all encouraging in terms of improving the usability of the smart home, but the industry still needs to ensure that those commands work every time. As many Alexa Plus and some early Gemini users have noted, the new voice assistants don’t always work with our old smart homes. Everyone admits this is a challenge. “LLMs are great at being creative, but not so good at doing the same thing over and over again with the same predictable output,” Kattukaran says. “That’s what the last generation of machine learning was reasonable at. The short-term gap for us is bringing the two together.”

This is why Google is launching two versions of Gemini: Gemini for Home, a more structured assistant that, while capable of understanding natural language, is designed specifically for the home. Then Gemini Live is a more creative, free-flowing chatbot that doesn’t require you to repeat the wake word. It will be available on select smart speakers with a subscription.

Unlike Alexa Plus, Gemini Live has no control over your smart home and can’t take any actions at all. Kattukaran says they plan to merge the two eventually, with Live’s capabilities being what he views as the future for smart home control. However, he cautions that with both Gemini versions, “there will be learning curves; it won’t be perfect on day one.”

Amazon took a different approach, going all in on its LLM for the home. Its Alexa Plus can control your home and have those free-flowing conversations. The company developed a way to connect the new LLMs with the more structured API pathways of the traditional smart home. “What the LLM is doing is thinking, making its round trips, making sure it calls the right API,” Panay says. “I think it’s our secret sauce. Nobody else is at this level.”

In practice, Alexa Plus doesn’t always do what it is told. He says the team is still working on fixing that disconnect, and it’s one of the reasons Alexa Plus is still in early access.

Reliably replacing the old with the new is the first hurdle these companies must overcome. In my testing of Alexa Plus so far, I’ve had several instances where things haven’t worked the way they used to. I’ve had to redo old routines because Alexa Plus doesn’t recognize them, and rename more obscure devices because it can’t parse the product name from a natural language command. (I have a smart mosquito repellent system called Liv, and when I ask it to “turn on Liv,” Alexa tries to show me a live stream of my cameras.)

The second hurdle is a heavy reliance on the cloud. During my testing of Alexa Plus and the brief demos I saw of Gemini for Home, the response times were noticeably slower than those of the original assistants, often taking upwards of 10 seconds.

Not everything is that slow — controlling Matter devices in Alexa, which connect locally, is faster, and Google Home should be the same. And Amazon said its new Omnisense platform can process some data locally on its Echo smart speakers.

But Alexa Plus works largely in the cloud. “I’m not worried about latency or security [from the cloud],” Panay says. “The team understands how to deliver the balance to our customers.” He added that he believes the cloud is the right solution. I disagree; a smart home that is reliant on an always-up internet connection is a hobbled smart home.

Finally, and perhaps the highest hurdle of all, they need to create a really compelling product that people will pay for.

It’s clear that both companies view generative AI as the long-awaited recurring revenue model for the smart home. Alexa and Google Assistant were free for the first decade of their existence, but these new versions require a Prime membership or a Google Home Premium subscription for most of Gemini’s advanced capabilities. (You can use the old Alexa and the natural language-capable Gemini for free.) Many of the flagship features also require a Ring or Nest subscription.

We’re a long way from something people will pay extra for. But what would tip the scale? Could a proactive AI assistant that not only controls your home but can also manage it for you provide enough value? Both companies are moving in that direction, and it will be interesting to see what they come up with.

“We’ve had this vision for a long time: can we deliver on the promise of having an AI security guard, an AI pet sitter, or an AI elder care?” says Kattukaran. “These are core needs in the home, and we think these components coming together start to deliver on that vision.”

Google’s new Home Brief, which can summarize all the actions your home has taken, lays the groundwork for a proactive system. Amazon’s new Omnisense platform could help Alexa understand more about a home’s occupants and react appropriately. Both companies’ smart security cameras are becoming virtual eyes for their AI assistants, thanks to AI-generated text descriptions and facial recognition technology.

I’d rather see less reliance on invasive cameras and more use of ambient sensing — millimeter wave, ultrawideband, ultrasound, Soli radar, RF sensing — although that’s far harder to implement. This is an area where companies like Apple, with its focus on privacy, and Home Assistant, with its commitment to local control, could step in with some compelling solutions.

Generative AI might be the key to unlocking a truly intelligent home, but there’s clearly still a long road to get there. And the race starts now.