Microsoft's Cortana Intelligence Institute cofounder on building better AI

The Cortana Intelligence Institute is a collaboration between Microsoft Research, Cortana Research and RMIT University in Australia, and it is focused on developing Cortana's next-generation capabilities.

I reached out to Ryen White, cofounder of the Institute and applied science manager who leads Cortana Research, for some insight. Here's what he had to say.

Related reading

• Cortana is headed to everyone, everywhere - if Microsoft gets its way
• Cortana's future is uncertain, but what about Microsoft's bot strategy?

Q&A with Cortana Research's Ryen White

Windows Central Writer Jason Ward: What and who initiated the idea of the Cortana Intelligence Institute? What barriers did you face to its establishment?

Ryen White: Cortana Intelligence Institutes were co-conceived by Roy Zimmermann, Director at Microsoft Research Outreach and me. This was part of discussions around strengthening Cortana ties with academia, providing an opportunity for academic colleagues to have industrial impact and driving product value for Cortana. The Microsoft Research and Cortana organizations have been extremely supportive of our efforts.

RMIT University is the "outsider" of this group. What is unique about RMIT that led Microsoft to reach out to them as a partner for the development of Cortana's next generation capabilities?

RMIT is a world-class research institution that greatly values applied research. This was a fabulous opportunity to work with experts in information retrieval, recommender systems and other disciplines at the university to enrich Cortana's capabilities.

Did you reach out to any other outside entities that will contribute in any way or declined the invitation?

This is the first year that we are running a Cortana Intelligence Institute and we are still testing and refining the concept. We issued an invitation for proposals to a select set of university research groups with relevant skills and expertise. We then reviewed proposals internally and selected RMIT. There are many university research groups worldwide doing Cortana-relevant research work and we would love to find a way to collaborate with them to bring value to Cortana.

What was Microsoft looking for in a partnership as it pushes Cortana forward?

We are primarily looking to develop innovative technologies for Cortana that build on the expertise and enthusiasm of university collaborators. We sought highly-capable research scientists who submitted proposals that address an important, high-impact research area, in this case centered around task intelligence.

What unique contributions do Microsoft Research and Cortana Research bring to the collaboration?

These organizations provide deep expertise in artificial intelligence and a path to product impact with the outcomes of the RMIT research.

What is the overall goal of this Institute?

The primary goal is to develop innovative technologies for Cortana, while also providing a way to strengthen relationships with academia and provide an opportunity for product impact for university researchers.

Is the institute a temporary entity that will at some point be dissolved? And can you describe what an "applied science organization" is?

Our first Cortana Intelligence Institute is slated to run for 18 months with the main objective of contributing new insights and innovative technologies to Cortana. We will evaluate the effectiveness of the institute and will decide about next steps in due course. Applied science organizations such as Cortana Research conduct research with the primary goal of realizing innovation within a product, in this case Cortana.

Will the institute's work complement other divisions and will other divisions affect it or will it exist more as an independent entity?

The institute will operate independently but will work closely with researchers in Cortana Research and Microsoft Research.

How many people are part of the Institute and what are the key skillsets?

There are around 15 researchers in this first institute across RMIT, Cortana Research, and Microsoft Research, including PhD students, post-doctoral researchers and more senior researchers. Skills include machine learning, information retrieval, recommender systems, human mobility and behavior analytics, spatio-temporal data mining and conversational search.

The institute's announcement reads, "The Cortana Research group is an applied science organization in Microsoft Cortana that was recently established to integrate research seamlessly into the digital assistant." What does that process look like?

Cortana Research is a research team situated inside the Cortana organization. We work with product owners to identify tough research challenges based on user needs and build technology to address these challenges for inclusion in future versions of the Cortana product.

How does that differ from other methods and from how rivals may integrate research into their digital assistants? What are the advantages?

The approach is seamless in the sense that the Cortana Research team is located within the Cortana organization. Such co-location is quite common in technology companies today. There are other approaches, such as having a separate research organization, which also has advantages. For the Cortana Intelligence Institute, we have both: we have the Cortana Research team located inside the Cortana product group and we are leveraging additional research expertise from Microsoft Research.

Are there any drawbacks that could result in "less tested" features making it into a product?

While having a research team inside the Cortana organization provides us with unique insight into what matters to Cortana users and a path to production, product teams also need to be agile. Our research investments can be impacted by an evolving product definition and the competitive landscape. While the Cortana Research team is flexible and focused on user/product needs, we also work hard to pick research problems that are broadly useful for Cortana and will stand the test of time.

Are ideas for Cortana's abilities conceived within the group, "pitched" to the company and then developed toward that goal? Or are ideas given to the group and then developed?

It really is a combination. We work closely with product managers to define the research necessary and how it fits with product goals. At the same time, we also explore a select set of research areas that are not in the immediate product plans, but they are important differentiators. Research at the institute is similar: it is driven by a combination of product needs and researcher vision.

Is the institute's role more abstract, where you research and develop the fundamental technologies that will enable Cortana's next generation abilities, and actual applications of that technology into practical feature sets is done by another entity?

The institute will tackle a mixture of problems that are important for Cortana right now while also having the freedom to explore next-generation capabilities. As the technology becomes ready, Cortana Research will help transfer it into the product.

Does your team work in collaboration with product roadmaps, or are you more an "independent resource" from which the Cortana product team draws?

Cortana Research works in close collaboration with the rest of the product team. That is a real advantage of having an organization like this be situated within Cortana. Although we perform our own research, we can also be a conduit for research from Microsoft Research and elsewhere. We tackle projects with a 6-12 month time horizon, ahead of where the product is now and shorter term than some of the research being conducted at Microsoft Research. Cortana Research has a portfolio of projects, some which are more product focused and some which are more speculative. As an applied research group, our projects are focused on improving Cortana.

Would the institute's role ever entail pushing a "breakthrough" for implementation in the product?

The primary goal is to have positive impact on Cortana. That can happen in many ways. The most common model will involve Cortana Research serving as first point of contact for research findings from the institute, directing those to the appropriate engineering teams in Cortana and working with those teams to ship the research outcomes.

Can you give a technical explanation of what has to happen to move from "query-based support", to being able to help with chores to having a rich dialogue with users. as was announced is the institute's goal for Cortana?

Cortana has a unique, holistic perspective across applications and devices. The key aspect of supporting tasks—from cooking a meal to arranging a meeting—is that they almost always require multiple steps. This is an area where Cortana can engage in a rich dialog with users and make suggestions to them about actions (skills) she can take to help them. In fact, we just announced skill chaining as a way for Cortana skill developers to connect skills to offer more comprehensive task completion support. Cortana Research is exploring contextual skill recommendation, an important enhancement to skill chaining that uses the current context to highlight just what Cortana can do.

Why is understanding context is so important?

A lot of the help that we require from digital assistants like Cortana depends strongly on the current context. Even location and time of day are important signals about the types of assistance that people are likely to require. For example, we see that users are more likely to request sleep sounds at home in the evening and productivity support, such as calendaring or email, at work during the day. Richer information about the current situation enables better support from Cortana.

Are there privacy or ethical concerns your team considers as you pursue your research? And what type of technology/sensors does RMIT's team hope to tie Cortana into so that she can "perceive" physical activity and location?

Privacy and ethical considerations are paramount. The project is going through full review by the RMIT Human Research Ethics Committee. Volunteers will consent to participate, and their data will be stored securely and shared only with project researchers. We will employ a range of sensors, including those from mobile devices, such as accelerometer, GPS and application usage, as well as ambient computing devices such as smart speakers.

Does the institute's research benefit Cortana on one form factor over another?

Task intelligence is a general concept that applies to many form factors. We want Cortana to help users achieve more while doing less across a range of devices and applications.

Do you or members of your team use Cortana? And is there was one thing that you'd love Cortana to do in the future?

Yes, we use Cortana often. I mainly use Cortana for music and reminders, including suggested reminders, a feature that I led the AI work on while I was in Microsoft Research. There are some great new Cortana features coming online that we have access to internally and will soon ship to all users. Looking ahead, I would like Cortana to be able to handle multi-step tasks that I do frequently – like setting up a meeting or booking travel – with a single voice command. Cortana should know my and others' preferences and act accordingly to save me time and effort.

Big thanks to Ryen White

The war between AI digital assistants is heating up as affordable smart speakers are becoming mainstream and assistants are learning more about us across devices. Microsoft's Cortana, though uniquely positioned as an enterprise and consumer product, lags behind the competition in mindshare, market presence and certain skills.

Cortana has a lot of ground to make up in the consumer space in order to compete with its rivals. But the Cortana Intelligence Institute, though focused on academia and the enterprise, may bring advancements to Cortana that will make her more competitive.