AI-boosted cameras help blind people to navigate

A prototype wearable system uses machine-learning to guide users around obstacles

A study participant tests out the device (left), which uses AI to interpret camera footage (top right) and give audio cues to help navigation.Credit: Tang et al., *Nature Machine Intelligence*

A system that helps visually impaired people to navigate their surroundings using cameras, earphones and artificial intelligence (AI) could offer advantages over white canes and other conventional technologies.

The system uses AI to interpret footage from camera mounted on a pair of glasses, and feeds the wearer information on their location in real time through audio alerts and vibration.

In a paper published in Nature Machine Intelligence on 14 April¹, researchers enlisted 20 people who have visual impairments to test out the devices. Participants’ walking distance and navigation time improved by 25% compared to using a cane when completing a 25-metre-long indoor maze.

“This system can partially replace the eyes,” says study co-author Leilei Gu, an AI researcher at Shanghai Jiao Tong University in China. Right now, the set-up is a prototype, he adds. To make it more useful on a practical level, the team needs to make sure that it’s very reliable, and that it’s safe for the user.

“This paper is about how to make a very, very intelligent stick,” says Botond Roska, director of the Institute of Molecular and Clinical Ophthalmology Basel in Switzerland. But he adds that it is “too early to say” to what extent people will actually adopt the technology.

Camera vision

The system features a pair of glasses equipped with a camera that captures live images of the wearer’s surroundings. A tiny computer processes these images using machine-learning algorithms trained to detect the presence of other people and objects such as doors, walls and furniture. The device gives the user audio cues about their surroundings every 250 milliseconds, producing a beep in either the right or left earphone to guide them in the right direction.

The researchers also created flexible ‘artificial skin’ patches that users wear on their wrists and fingers, for additional navigational assistance. The bracelets are equipped with their own camera system and the patches vibrate to alert the wearer when an obstacle is between 40 centimetres and 5 centimetres away. They also vibrate when the user reaches for an object, to alert them when it’s time to grasp it.

Such smart technologies might be more helpful than conventional sticks for people living in large cities, says Roska. Cameras have a higher range than a stick, which can typically detect only obstacles that are around one metre in front of the person, says Gu. “It can only touch the environment, it cannot know what the object is,” he adds.

The cameras are currently mounted on glasses, but the team is working on making the devices lighter and more discreet.Credit: Tang et al., *Nature Machine Intelligence*

The researchers first worked with 12 participants with visual impairments to evaluate how well the system could help them to avoid obstacles when walking through rooms. After training with the system, all participants were able to navigate indoors, and their walking speed was similar to when they used a cane.

In another trial, the technology continued to perform well when eight participants tested the device in real-world situations — including walking on a city street and through a conference room cluttered with furniture.