Text extraction and retrieval from smartphone screenshots

Abstract

Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74% character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.

A Robust Algorithm for Emoji Detection in Smartphone Screenshot Images

“…The widespread use of smartphones to capture, share, and store texts and attached objects elicits the interests of researchers on smartphone-based images. Many studies have considered smartphone images [3][4][5][25][26][27][28][29][30][31][32][33][34]. State-of-the-art methods for smartphone images are limited to text recognition and ignore images.…”

“…State-of-the-art methods for smartphone images are limited to text recognition and ignore images. Surveys on screenshot textual images were conducted in [4] and [5].…”

“…The smartphone is assumed to be the most used device in the present era [1,2]. The wide use of smartphone devices has resulted in the production of a huge number of digital images captured by cameras or screenshots [3,4]. These digital images provide a hybrid collection of graphical and textual information, which require separate methods that can handle these types of contents [5].…”

“…Chiatti, et al [4] claim that the increment in data extraction from screenshots increases the interest of individuals in methods that could address and extract data from various texts, fonts, graphical contents, and layouts.…”

On-Device Information Extraction from Screenshots in form of tags

“…However, it poses a di erent set of challenges like cooccurrence of icons and text and a vast set of fonts and layouts [18]. Further, screenshots represent a hybrid case study, mixing graphic and scene text in varying proportions over time, hence motivating the evaluation of existing techniques on a novel collection [3]. e work presented in [3] shares some similarities with our current pipeline, however its more focused on extracting complete OCR, which is language dependent and server-based.…”