How Microsoft's Cortana is getting her groove back

Microsoft's Cortana has been undergoing an identity crisis in the last few years. As Windows Phone wound down, Microsoft tried, in vain, to push Cortana as a consumer option with the Invoke speaker and GLAS thermostat. It all failed miserably.

But Microsoft's ambitions for smart AI and Cortana seems to now be finding its groove. The company recently unveiled Play My Emails for Outlook Mobile along with some significant enhancements to Outlook calendar.

Could this new Cortana finally be the path forward for Microsoft? We sat down with Andrew Shuman, Corporate Vice President of Cortana, to talk about why putting the AI in Outlook makes sense and how Cortana is finally going to work.

Cortana finds purpose in Outlook Mobile

This week Microsoft announced the new Play My Emails feature for Outlook mobile which uses a new neural text to speech engine with more natural intonation and pauses. We first reported on this feature in late 2018.

Currently on iOS (beta) and coming to Android in early 2020, the idea sounds like simple speech-to-text, but it is much more advanced. For instance, Cortana won't read unnecessary parts of the email or things like shipping numbers. Instead, it smartly figures out the actual core message and narrates it back to you. This is machine learning in action.

If the email is threaded with multiple responses, Cortana will tell you that, pausing between emails, announcing who is responding, and walking you through the processes.

The idea for Play My Emails borrows heavily from podcasts, but also software for the visually impaired. The Outlook team spent a month wearing goggles that impair vision as well as studying screen readers to understand better how those systems work. It's a great example of how technology meant to help those at a disadvantage can trickle down for everyone.

Imagine closing your eyes and giving your phone to someone and asking them to read your emails. Cortana prefaces each session with the number of emails, how long it will take, and information like "this is a long one." If there are photos in the email, Cortana will tell you that, too, suggesting you will need to open the app later to see them yourself (one could imagine someday AI will even be able to describe the photo as well).

For Microsoft 365 customers, a new calendar feature lets you add Cortana to your email. If you need to schedule an appointment with someone via email, you can add Cortana as one of the recipients. Cortana then uses the company's directory and internal calendar to suggest times that work for both of you. If the other person is not part of your org, they can email back using natural language the time and date, and Cortana will handle the rest of the scheduling.

This ability is a significant feat for AI. It isn't an agent responding to you with an answer; it's going out and doing something on your behalf. Shuman notes this scheduling ability is "less of an interactive function and more of a delegate one." It's also the sort of task a real assistant would do at the office.

Another innovative feature is one that is also pro-active. Cortana serves up a daily briefing within Outlook that suggests emails you need to respond to, tasks you need to finish, appointments coming up, and more. Because Microsoft has been adding "app-like" features to Outlook, users can hit actionable buttons within the email to complete those tasks.

Cortana can even suggest a few moments throughout your day for focus time based on your schedule. If a suggested time works, hit a button within the daily brief, and your calendar is blocked off so you can get work done. Shuman refers to this as "Cortana as a coach."

All these features have one thing in common: productivity. It may seem trivial, but these are the kind of lofty goals for artificial intelligence. Assisting people with emails, schedules, making appointments, finding downtime, suggesting responses, and more is what real technology should be doing — merely telling you the depth of the Grand Canyon is easy by comparison.

None of this ignores Microsoft's other ongoing issues with Cortana, like regional availability and support for other languages. Indeed, many of these new Cortana features are US-only, a familiar theme. Shuman notes the team still wants to have Cortana in other regions, but for now, not much has changed, and that calls into question how well this new strategy will work.

Putting Cortana where it makes sense

The most significant change for Microsoft's thinking on Cortana comes down to where the assistant lives. The idea of a centralized "hub" explicitly made for Cortana on iOS and Android is making less sense these days (the depreciation of features and functions bolster this argument). The Microsoft Surface Headphones and Surface Earbuds are also getting dedicated apps for iOS and Android nullifying the need for users to install Cortana. It seems highly likely those dedicated mobile apps will go away sometime in 2020.

Reports that Microsoft Launcher too may be ditching Cortana also seems probable. Even Xbox may lose its Cortana skill.

All of that seems to suggest Cortana is, wait for it, dead. But Andrew Shuman sees it differently. It is about focusing on specific problems and going to where it already has an audience. Regarding this week's announcements and the future of mobile, Shuman notes:

The argument is a strong one. Cortana cannot compete against an e-commerce company like Amazon or Google's Android, where its assistant handles phone and mapping functions. But Microsoft can play to its strengths. Outlook Mobile has millions of users on Android and iOS — why not put an assistant there? Why not use the power of AI to help manage, triage, and even act on all that data?

It's not a rhetorical question either. Up until now, digital assistants have been a lot of novelty (jokes, riddles, trivia) with limited functionality (play music, set a reminder). What Alexa, Google, and Siri have not done is become actual assistants — ones that proactively solve problems. That's the real goal of AI, after all, to make your life easier, to help you manage things, not offer novel distractions or entertainment.

Another place where Cortana still makes sense for Microsoft is being front-and-center for Windows 10 (no, it is not going anywhere). New features, including previously announced conversational style and semantic awareness, are nearing completion. Weaving a digital assistant and AI throughout Windows and its services is still a priority for Shuman and his team, and we can expect more on that front in 2020.

Also, don't expect a rebranded "Microsoft Assistant" in place of Cortana. When pressed on the subject of Cortana as an agent Shuman says that Cortana — the name and likeness — is sticking around. When the assistant reads back your email in Outlook Cortana's blue ring is front and center, same when CC'ing Cortana on an email. Shuman notes that they will use Cortana, where cross-device experiences make sense. Other times like in PowerPoint, where AI assists in slide preparation, it is the same tech powering Cortana, but with no visible representation.

Cortana with focus, not tricks

It's hard not to conclude that Microsoft is pushing Cortana towards a more niche, but highly functional area within its core products. The targets are enterprise-first right now with less focus on the consumer space, but Shuman notes that these features lend themselves to that area too.

Partnerships with Amazon's Alexa can pick up Cortana's slack for smart-home functions. Shuman's belief in a multi-client world seems more to be out of necessity, but that doesn't make it the wrong choice. The Voice Interoperability Initiative with partners like Salesforce, Sonos, Sony Audio Group, Spotify, Verizon, Orange, and Qualcomm suggest this could be a more successful approach to AI and digital assistants than a monolithic model. Shuman remarks that in the real world, you have experts and professionals for health, fitness, work, entertainment, and more. Digital assistants should look like that too.

Microsoft's mantra since CEO Satya Nadella's takeover in 2015 is about getting back to what Microsoft is good at, namely software and services. With Cortana playing to Microsoft's strength in Outlook, Teams, Office, Bing Search, and more, it is one space that the company can have a significant impact.

All of this may disappoint those who want a "one AI to rule them all" world with Microsoft at the helm, but such a scenario is seemingly more unlikely by the day. Cortana may ditch some of these haphazard experiments like Xbox, home speakers, and phones, but this new direction seems more realistic.

If Cortana was a kid in 2014, it is now headed off to the real world in 2019 armed with some real AI under its belt. It's time for the assistant to do more than answer trivia or play your favorite diddy. It's time to get to work.