[ad_1]
Topic: Disability Visual Aids
Author: University of Michigan
Published: 2024/10/11
Publication Type: Reports & Proceedings
Contents: Summary – Introduction – Main Item – Related Topics
Synopsis: A world of color and texture could soon become more accessible to people who are blind or have low vision through new software called WorldScribe. The tool uses generative AI (GenAI) language models to interpret the camera images and produce text and audio descriptions in real time to help users become aware of their surroundings more quickly.
Why it matters: This article introduces WorldScribe, an innovative tool developed by University of Michigan researchers that could dramatically improve the daily lives of people who are blind or have low vision. WorldScribe uses generative AI to provide real-time audio descriptions of surroundings captured by a camera, offering unprecedented access to visual information for those with visual impairments[2]. The tool’s ability to adjust detail levels, adapt to noisy environments, and respond to user queries demonstrates its potential to enhance spatial awareness and independence for blind individuals. By providing immediate, comprehensive descriptions of the environment, WorldScribe could reduce the mental effort required to understand surroundings, allowing users to focus more on interacting with the world around them – Disabled World.
Introduction
A world of color and texture could soon become more accessible to people who are blind or have low vision through new software that narrates what a camera records. The tool, called WorldScribe, was designed by University of Michigan researchers and will be presented at the ACM Symposium on User Interface Software and Technology in Pittsburgh next week.
Main Item
The tool uses generative AI (GenAI) language models to interpret the camera images and produce text and audio descriptions in real time to help users become aware of their surroundings more quickly. It can adjust the level of detail based on the user’s commands or the length of time that an object is in the camera frame, and the volume automatically adapts to noisy environments like crowded rooms, busy streets and loud music.
Continued…
The tool will be demoed at 6 pm EST Oct, 14, and a study of the tool-which organizers have identified as one of the best at the conference-will be presented at 3:15 pm EST Oct. 16.
“For us blind people, this could really revolutionize the ways in which we work with the world in everyday life,” said Sam Rau, who was born blind and participated in the WorldScribe trial study.
“I don’t have any concept of sight, but when I tried the tool, I got a picture of the real world, and I got excited by all the color and texture that I wouldn’t have any access to otherwise,” Rau said. “As a blind person, we’re sort of filling in the picture of what’s going on around us piece by piece, and it can take a lot of mental effort to create a bigger picture. But this tool can help us have the information right away, and in my opinion, helps us to just focus on being human rather than figuring out what’s going on. I don’t know if I can even impart in words what a huge miracle that truly is for us.”
During the trial study, Rau donned a headset equipped with a smartphone and walked around the research lab. The phone camera wirelessly transferred the images to a server, which almost instantly generated text and audio descriptions of objects in the camera frame: a laptop on a desk, a pile of papers, a TV and paintings mounted on the wall nearby.
The descriptions constantly changed to match whatever was in view of the camera, prioritizing objects that were closest to Rau. A brief glance at a desk produced a simple one-word description, but a longer inspection yielded information about the folders and papers arranged on top.
Continued…
The tool can adjust the level of detail in its descriptions by switching between three different AI language models. The YOLO World model quickly generates very simple descriptions of objects that briefly appear in the camera frame. Detailed descriptions of objects that remain in frame for a longer period of time are handled by GPT-4, the model behind ChatGPT. Another model, Moondream, provides an intermediate level of detail.
“Many of the existing assistive technologies that leverage AI focus on specific tasks or require some sort of turn-by-turn interaction. For example, you take a picture, then get some result,” said Anhong Guo, an assistant professor of computer science and engineering and a corresponding author of the study.
“Providing rich and detailed descriptions for a live experience is a grand challenge for accessibility tools,” Guo said. “We saw an opportunity to use the increasingly capable AI models to create automated and adaptive descriptions in real-time.”
Because it relies on GenAI, WorldScribe can also respond to user-provided tasks or queries, such as prioritizing descriptions of any objects that the user asked the tool to find. Some study participants noted that the tool had trouble detecting certain objects, such as an eyedropper bottle, however.
Rau says the tool is still a bit clunky for everyday use in its current state, but says he would use it everyday if it could be integrated into smart glasses or another wearable device.
The Research
The researchers have applied for patent protection with the assistance of U-M Innovation Partnerships and are seeking partners to help refine the technology and bring it to market.
The research was funded by U-M.
Guo is also an assistant professor of information within U-M’s School of Information.
Study: WorldScribe: Towards Context-Aware Live Visual Descriptions
Attribution/Source(s):
This quality-reviewed publication was selected for publishing by the editors of Disabled World due to its significant relevance to the disability community. Originally authored by University of Michigan, and published on 2024/10/11, the content may have been edited for style, clarity, or brevity. For further details or clarifications, University of Michigan can be contacted at umich.edu. NOTE: Disabled World does not provide any warranties or endorsements related to this article.
Page Information, Citing and Disclaimer
Disabled World is a comprehensive online resource that provides information and news related to disabilities, assistive technologies, and accessibility issues. Founded in 2004 our website covers a wide range of topics, including disability rights, healthcare, education, employment, and independent living, with the goal of supporting the disability community and their families.
Cite This Page (APA): University of Michigan. (2024, October 11). WorldScribe: Real-Time Surrounding Descriptions for the Visually Impaired. Disabled World. Retrieved October 11, 2024 from www.disabled-world.com/assistivedevices/visual/worldscribe.php
Permalink: <a href=”https://www.disabled-world.com/assistivedevices/visual/worldscribe.php”>WorldScribe: Real-Time Surrounding Descriptions for the Visually Impaired</a>: A world of color and texture could soon become more accessible to people who are blind or have low vision through new software called WorldScribe.
Disabled World provides general information only. Materials presented are never meant to substitute for qualified medical care. Any 3rd party offering or advertising does not constitute an endorsement.
[ad_2]
