According to the World Health Organization, an estimated 2.2 billion people across the globe are visually impaired. 216.6 million have moderate to severe visual impairment, and 36.0 are blind (The Lancet: Global Health, 2015). In 2016, Microsoft launched what is known as Project Tokyo, a collaboration between researchers from the United States, the United Kingdom, China, Japan, and India to develop life changing technology for those with visual impairments. The result is a specialized Microsoft HoloLens designed to help the visually impaired interact with people around them.
With the lens removed, a set of cameras tracks the person closest to the wearer, and a strip of LED’s flashes green when the person has been identified to let them know they have been seen. The algorithms can map where people are in a room, and if a person’s name has been learned into the system, the headset will identify them.
This information is translated to the wearer via different audio cues. The device is rigged with accurate spacial surround sound that can imitate clicks coming from specific places. For instance, if a person is standing six feet to the right, the wearer will be informed by clicking noises that sound as though they are coming from where that person is standing. If the system recognizes a face, it sounds a “bump” and will identify any person it knows the name of. In order to guide the wearer’s eyes toward a person’s face, it plays a sound like stretching rubber bands followed by a click when the camera focuses on a person’s nose. If a person is looking at the wearer, the system chimes.
While the project is still in the works, Microsoft is already using this modified HoloLens to help visually impaired children develop social skills. It is used alongside Microsoft’s mobile apps SeeingAI and Soundscape which are designed to help low vision users map the world around them.