When AI assistants first hit the market, they were far from ubiquitous, but thanks to more third-party OEMs jumping on the smart speaker bandwagon, there are more choices for assistant-enabled devices than ever. In addition to increasing variety, in terms of hardware, devices that support multiple types of AI assistants are becoming more common. Despite more integration, competition between AI assistants is still stiff, so to save you time and frustration, we did an extensive hands-on test – not to compare speakers against each other, but to compare the AI assistants themselves.
There are four frontrunners in the AI assistant space: Amazon (Alexa), Apple (Siri), Google (Google Assistant) and Microsoft (Cortana). Rather than gauge each assistant's efficacy based on company-reported features, I spent hours testing each assistant by issuing commands and asking questions that many business users would use. I constructed questions to test basic understanding as well as contextual understanding and general vocal recognition.
Accessibility and trends
Accessibility refers to how many devices the virtual assistant lives on. The best way to measure these AI assistants against each other in this category is to consider them individually. They are listed from most accessible to least accessible.
Google Assistant
Google Assistant is available on all Android and iOS devices as well as on Chromebooks. Google has its own line of Google Home speakers, including Google Home Mini ($49), Google Home ($129) and Google Home Max ($399), all of which are built for use with Google Assistant. Google also makes it easy to search third-party Google AI-enabled speakers, like the Harman JBL Link 10 ($99.99), Sony LF-S50G ($199.99), Panasonic SC-GA10 ($249.99), Onkyo VC-GX30 ($199.99), Mobvoi TicHome Mini ($79.99), LG ThinQ WK7 ($199.99) and Best Buy Insignia Voice Smart Bluetooth Speaker ($59.99). The fact that so many major players in the speaker and headphones space are releasing Google AI-enabled hardware at varying price points speaks volumes as to who is coming out ahead in the AI assistant space.
Cortana
Microsoft's AI, Cortana, comes standard on Windows machines, but it's also available for download on Android and iOS. Cortana is available on the $199.95 Harman Kardon Invoke with Cortana speaker, and while there have been rumors of Cortana-enabled HP and Xiaomi speakers for quite a while, there's no concrete evidence that such speakers are still in the works. Whether Cortana will be available on other third-party speakers or not remains to be seen, but it looks as though the Microsoft AI assistant is losing traction in the face of more sophisticated competition, at least for the consumer market.
At Microsoft Build 2018, the tech giant unveiled its long-promised integration with Amazon Alexa. The partnership between Amazon and Microsoft suggests it may be early to count Cortana out of the game entirely, especially if it continues to focus primarily on the enterprise market rather than home use.
Alexa
Amazon's smart assistant is accessible through the Amazon Echo line of speakers as well as through Amazon's line of Fire tablets and Fire TV. Of all the AI assistants, Alexa is available on the widest variety of devices in terms of design. While Google's AI assistant can be found on plenty of speakers, they all look similar. Amazon, on the other hand, sells dozens of Alexa-enabled designs, including speakers of all sizes as well as modern-looking speakers with glossy touchscreens.
There are also some third-party smart speakers and other devices that support Alexa, including the Sonos wireless speaker ($199.99), Ultimate Ears Megablast speakers ($199.99), Altec Lansing VersA Smart Portable speakers ($99.99) and the Garmin Speak Plus Dash Camera ($179.99). There is an Alexa app as well, but the app is intended primarily as a supplement to another Alexa device and not as a stand-alone AI assistant.
Siri
You can access Siri on nearly any Apple device, including its line of laptops, desktops, phones and tablets, and smartwatches. Apple also sells its own speaker, the HomePod ($349), and it looks like third-party access is on the way. Sonos, a company that produces and sells high-end home speakers, claims it will roll out a software update in July that will allow users to access Siri (in addition to Alexa and Google Assistant).
Ease of setup
Ease of setup refers to how long it takes to get the assistant up and running, as well as how complicated that process is. All the AI assistants I evaluated have options to customize the personal assistant experience, including app integration, custom settings and adaptive responses.
To maintain an even testing ground, I did not consider additional setup steps that fall under the category of customization; I only considered how long it would take a new user to turn on the device and start asking questions. Google Assistant, Siri and Alexa all required virtually no setup time (you just sign in to a network and start) and were completely intuitive. None of these three took more than about 45 seconds for initial access, and that included turning the device on or, in Alexa's case, plugging it in.
Cortana is a different story. If you want to access the voice features of Cortana, which is what this test was all about, you must access it on your device and then log in – oh, and you can't use a work or school email address for the login. Once you enter an acceptable email account, wait for a verification code and enter the code, you think you're in business, but you're not.
Ultimately, I successfully logged in by downloading the Cortana app and signing in on my phone. But when you do that, Cortana won't allow you to sign in with your phone number (even though it claims that's an option) because it's "not a Windows account." You can't sign in with your name either, just with your non-work and non-school email address.
After I signed in with my email address, Cortana cheerfully addressed me by that, my entire email address (dot-com included), for the remainder of our conversations, but at least it started working. Hopefully, the setup process will be easier for the Harman Kardon speakers that will feature Cortana when they finally debut.
Voice recognition
For voice recognition, all I wanted to know was how often the virtual assistant could recognize the words I was saying. I didn't consider context or the value of the response I was given, just basic recognition. I tested voice recognition at various distances from the devices, with varying levels of background noise.
Google and Siri understood me well when the room I was in was quiet and I was close to the devices. There were a couple funny misunderstandings, like when I asked Siri, "What's the date four weeks from now?" and it gave me the date for one week later, because it confused "four" with "for" and "weeks" with "week" and thought I was saying, "What's the date for a week from now?" All in all, though, the voice recognition was impressive under ideal conditions.
However, once I used Google or Siri to do anything sound-related, like read a news article or play music, the assistants could no longer hear me speaking at a normal volume. I literally shouted, and eventually, Google heard me, but Siri didn't hear me even then. In fact, I had to manually turn off the sound on the iPhone to get Siri to stop playing the news, which kind of defeats the hands-free purpose.
Alexa's voice recognition was spotty. Like Google and Siri, it could not understand me at all when there was even soft music playing, and I had to manually shut down the speaker to get it to stop playing music. Unlike the other two assistants, Alexa also had issues understanding basic questions, even when the room was silent except for my voice commands.
I repeatedly asked for help getting plane tickets, and each time Alexa thought I was asking for movie tickets and directed me to Fandango. When I asked for future dates ("What's the date for a week from now?"), it simply replied, "Sorry, I don't know that," while other assistants had no trouble understanding identically phrased questions.
Cortana performed the worst by far in basic voice recognition. Microsoft's assistant had issues understanding me even with zero noise interference. Here are just a few examples of basic inquiries Cortana could not understand, even when stated slowly and clearly with no background noise:
- Cancel this task.
- I want to set a reminder.
- Does Amazon sell printer paper?
- I need help finding a restaurant.
- Do I have any reminders coming up?
The other three assistants could understand all these inquiries, and Cortana understood some of the inquiries I made (like those about the weather), so it wasn't a microphone issue.
Success of queries and ability to understand context
The value of an AI assistant lies in its ability to understand natural language and context and deliver a useful response. To test this skill, I devised questions with context-reliant follow-ups. Here are a few questions I asked each of the assistants:
- How much is $5 in euros? What about in yen?
- Where is there an Applebee's near me? Can you make a reservation?
- How long will it take me to get to LaGuardia Airport in the car? How about by subway?
I also asked each of the assistants a laundry list of common questions about scheduling, setting reminders, online shopping, booking travel accommodations and getting directions.
Cortana
Unfortunately, Cortana effectively removed itself from the running because it couldn't even hear or understand me on a basic level. When Cortana did understand me, it did not respond in a way that was intuitive or helpful. For example, when I asked for directions to the nearest airport, it searched for nearly a full minute and then returned with a list of results from Bing, which mostly linked to general airline ticket websites such as Expedia.
When I asked Cortana to translate from USD to yen, which all the other three assistants did with ease, it did not reply aloud but rather brought up a page from Bing with a list of general responses about currency exchange from Answers.com. Anytime I asked follow-up questions, especially those dependent on context, Cortana either didn't respond at all, told me it didn't know or directed me to allow permissions from a third-party app.
While you can integrate third-party apps with Cortana (as you can with all the assistants), my guess is that the process for doing so is long-winded at best. In general, Cortana seems like a way to Bing things aloud and then receive answers on your laptop, rather than a full-fledged virtual assistant for business.
Alexa
Alexa worked well when answering basic questions, especially those that pertained to purchasing items on Amazon and setting reminders. However, when I worked my way down the list and got to more complicated questions, which required context or close attention to detail, Alexa faltered. I attempted multiple times to get Alexa to help me purchase airline tickets, but the assistant consistently referred me to Fandango to buy movie tickets. It rarely answered follow-up questions at all, and when it did, it was often with a polite but unhelpful "Sorry, I don't know that."
In general, Alexa could do part of each task I asked of it. For example, it could tell me where the closest Applebee's was, but it couldn't make a reservation. It could tell me how much $5 was in euros, but when I followed up with "What about yen?" it didn't know. When I asked how long it would take for me to get to LaGuardia Airport, it said, "As I don't know your speed, I can't tell you how long that will take."
Alexa seems to rely on very specific terminology for commands. For example, when I asked it to read me the news, it did, but I couldn't get it to stop. I tried every possible direct command I could think of to get it to stop reading the news, including "stop" and "turn volume off," but the only phrase that worked (which I had to look up online) was "stop flash briefing," which isn't exactly intuitive. It was easy to set up reminders with Alexa, but when the reminder time would come, Alexa would just blare an alarm, with no indication of what the alarm was for, and I had to turn off the alarm manually.
Alexa is an OK tool for ordering on Amazon, and it may be good for smart house integration, but it's not the best AI for natural language use. Anyone adopting the Alexa should peruse user guides before setting it up and understand its limitations.
Google Assistant vs. Siri
It's worthwhile to compare Google Assistant's and Siri's ability to understand the context and intuitive speech because these two assistants are neck and neck in ease of use and intelligence.
When I asked for directions to One World Trade Center, both Google Assistant and Siri responded with clear driving instructions. When I added a follow-up question about public transportation directions instead, Google Assistant responded verbally with directions to and from the subway, as well as an automatic link to step-by-step instruction in Google Maps. Unfortunately, Siri simply found me alternate driving directions. No matter how I asked, I could not get Siri to give me public transit directions.
Siri bested Google Assistant in tasks like finding specific restaurants and making reservations. In fact, Siri was the only assistant I tested that could not only find a nearby restaurant (which Google Assistant did too) but also place a reservation for me. When I asked Google Assistant to find a specific restaurant, it did, but when I asked it to book a reservation for me, it inexplicably took me to DisneyWorld.com. I tried several times to book a reservation through Google Assistant, and it just wasn't happening. Incidentally, it also failed to convince me to book a Disney vacation.
The ability to make hands-free restaurant reservations with Siri could be a real boon to business users who want to use their AI assistants for business and personal use. However, it falls short if you are not specific in your requests. For example, when I simply said, "I need help finding a restaurant," Siri responded with a list of Google results that were literally instructions on how to find a restaurant, while Google Assistant read between the lines and showed me restaurants nearby.
Google Assistant was also better than Siri for travel information, but it still wasn't perfect on follow-through. When I asked Siri for flights from New York to Paris, it just Googled my question and showed me the responses. When I asked Google Assistant for flights from New York to Paris, it asked me for dates and started pulling up available reservations; however, when I tried to go all the way and book the flights, it simply said, "No problem!" and then did nothing. One cool thing Google Assistant offered me, without prompting, was the ability to email me if prices changed on the flights I was interested in.
When I asked the assistants to read me the news, Google Assistant immediately started playing a recent NPR podcast, while Siri Googled several news sources for me. I noticed throughout many different tasks that Siri rarely responded to me aloud, while Google Assistant nearly always did. While this is likely a matter of playing with a few settings, it's an interesting difference to note.
Bottom line
None of the AI assistants are perfect; this is young technology, and it has a long way to go. There was a handful of questions that none of the virtual assistants on my list could answer. For example, when I asked for directions to the closest airport, even the two best assistants on my list, Google Assistant and Siri, failed hilariously: Google Assistant directed me to a travel agency (those still exist?), while Siri directed me to a seaplane base (so close!).
Judging purely on out-of-the-box functionality, I would choose either Siri or Google Assistant, and I would make the final choice based on hardware preferences. None of the assistants are good enough to go out of your way to adopt. Choose between Siri and Google Assistant based on convenience and what hardware you already have.