What you need to know
- A new post from Microsoft breaks down how Project Tokyo helps blind people recognize faces.
- The project uses artificial intelligence and specialized hardware to help send cues and information to blind people.
- The post focuses on a 12-year old boy, Theo, and the growth of Project Tokyo.
Microsoft continues to use technology to improve accessibility. A new blog post from Microsoft shares the growth and development of Project Tokyo, a combination of technologies that extend people's capabilities. Project Tokyo uses artificial intelligence, specialized hardware such as a custom HoloLens, and the results of several research efforts to help improve accessibility. One of the key areas that the post highlights is the ability to send spatial audio cues and information to blind people and people with low vision.
The post focuses heavily on the story of Theo, a 12-year old boy who uses a customized HoloLens to improve his communication. The headband detects people's faces, and if a face is on the system, sends an audio cue to Theo to tell who who he's facing. The post explains that some blind people can miss cues that people with full vision take for granted, such as knowing when someone is making eye contact with them. The system Theo uses helps communicate those cues and gives Theo and others an improved sense of connection.
Theo states that "It is so exciting to be able to find out where the people are in my environment," adding, "Not just who chooses to talk, but all of the people who are silent that you can see by their face, but I can't."
Project Tokyo isn't restricted to vision-related aspects of communication. Researcher Cecily Morrison views the future of AI as a set of resources that people can use in a variety of ways, "All of a sudden we don't have to say, 'Hey you are blind and I just made this accessible to you.' We say, 'Hey, you are you and I have just built a system that works for you.'"
Proof that Microsoft's Windows eye-tracking tech works: This article was written with it
Project Tokyo came from a challenge in 2016 to create AI systems that could do more than simple tasks. The blog post shares the story of how a team from Microsoft followed athletes on a trip to the 2016 Paralympic Games in Rio de Janeiro, Brazil. The trip helped Microsoft's team understand communication and group dynamics affected by blindness and vision impairments.
Ed Cutrell, a senior principal researcher at Microsoft, states, "We, as humans, have this very, very nuanced and elaborate sense of social understanding of how to interact with people – getting a sense of who is in the room, what are they doing, what is their relationship to me, how do I understand if they are relevant for me or not... And for blind people a lot of the cues that we take for granted just go away."
People working on Project Tokyo realized that the HoloLens already picks up a lot of the information that they needed in a system. It has a wide field of view, can recognize faces, and has speakers that support spatial audio. These features helped make a system that can identify people, contextually draw people's attention to a location with spatial audio, and to deliver cues to improve communication.
How Microsoft's Xbox Adaptive Controller changed the game for one man with disabilities
In addition to improving communication directly, Microsoft is looking into if Project Tokyo can help teach blind children or children with low vision to improve interactions.
Microsoft plans to expand Project Tokyo in the future. Its goal is to be able to adapt the system to different situations and preferences.
