Evaluating the effectiveness of automatic image captioning for web accessibility

Abstract

The web has become a fundamental tool for carrying out many activities spanning from education to work and private life. For this reason, it must be accessible to every user regardless of any form of impairment or disability. Images on the web are a primary means for communicating information, and specific HTML elements were defined to enrich images with textual descriptions, which can be read aloud by screen readers or rendered by braille displays. A relevant problem is that adding a text describing each image published on a website is a demanding task requiring a non-negligible effort for web developers. Several tools based on machine learning have emerged, which can automatically return descriptions for the images. In this work, we evaluate the correctness of their outputs by comparing the generated descriptions with human-defined references. More specifically, we selected 60 images from Wikipedia and their corresponding descriptions as defined by Wikipedia contributors. We then generated the corresponding descriptions employing four state of the art tools (Azure Computer Vision Engine, Amazon Rekognition, Cloudsight, and Auto Alt-Text for Google Chrome) and asked 76 computer science students to blindly evaluate the perceived correctness of the descriptions without being aware of their source. The results show that the descriptions available in Wikipedia are still perceived as the best ones. However, some tools generate good results for specific categories of images, and they can represent proper candidates for the automated and massive addition of image descriptions to websites, helping to increase the accessibility level of the web drastically.

ShoppingTotal: A Mobile Application Utilizing Assisted Rekognition Algorithm for Intelligent Price Detection from Shelf Label Images

“…Amazon Rekognition has several other features for object detection and image classification. Studies on Amazon Rekognition include image tagging (Kuang et al, 2021), face identification (Ali et al, 2022) (Liu & Wilkinson, 2020), emotion detection (Yang et al, 2021), and image captioning (Leotta et al, 2022).…”