Apple accessibility engineers discuss new Screen Recognition feature on iOS 14

Apple introduced new ‘Screen Recognition’ technology on iOS 14 which automatically labels every button, tab, and slider to make it easier for blind users to operate an iPhone. Recently, Apple engineers sat down with TechCrunch to give a detailed insight into the company’s approach towards improvements in accessibility features.

Apple’s accessibility features in all its devices have helped individuals with visual and hearing impairments, for years, to not only operate the devices but also to easily carry out complex tasks like sending emails, web browsing, composing music, and more.

A new AR-enhanced ‘People Detection’ feature was released on iOS 14.2 and Apple Support app was updated with improved accessibility along with other features. VoiceOver is already an acclaimed Apple accessibility feature that enables musicians, lawyers, activists, and entrepreneurs with disabilities to carry on their work.

Engineers discuss improvements in Apple Accessibility features

Talking about improvements in Apple accessibility features from iOS 13 to iOS 14, Chris Fleizach, iOS accessibility engineer, and Jeff Bigham, AI/ML team member explained the philosophy behind the new Screen Recognition labeling technology. They said,

“We looked for areas where we can make inroads on accessibility, like image descriptions. In iOS 13 we labeled icons automatically – Screen Recognition takes it another step forward. We can look at the pixels on screen and identify the hierarchy of objects you can interact with, and all of this happens on device within tenths of a second.

It was done by taking thousands of screenshots of popular apps and games, then manually labeling them as one of several standard UI elements. This labeled data was fed to the machine learning system, which soon became proficient at picking out those same elements on its own.”

Apple accessibility

Bigham said that although the idea of Screen Recognition is not new, it has been perfected now due to advancements in technology.

“VoiceOver has been the standard bearer for vision accessibility for so long. If you look at the steps in development for Screen Recognition, it was grounded in collaboration across teams — Accessibility throughout, our partners in data collection and annotation, AI/ML, and, of course, design. We did this to make sure that our machine learning development continued to push toward an excellent user experience.”

This new intuitive capability will have a more profound impact by making apps more accessible for users with vision impairments. Currently, the new Screen Recognition capability is available on iOS and might be released on Mac in the future.

Read Also:

About the Author

Snapchatter and Apple fangirl, covers everything about social media, app reviews and Apple news.