FlexType: Flexible Text Input with a Small Set of Input Gestures

Mobile apps have become indispensable for accessing and participating in various environments, especially for low-vision users. Users with visual impairments can use screen readers to read the content of each screen and understand the content that needs to be operated. Screen readers need to read the hint-text attribute in the text input component to remind visually impaired users what to fill in. Unfortunately, based on our analysis of 4,501 Android apps with text inputs, over 76% of them are missing hint-text. These issues are mostly caused by developers' lack of awareness when considering visually impaired individuals. To overcome these challenges, we developed an LLM-based hint-text generation model called HintDroid, which analyzes the GUI information of input components and uses in-context learning to generate the hint-text. To ensure the quality of hint-text generation, we further designed a feedback-based inspection mechanism to further adjust hint-text. The automated experiments demonstrate the high BLEU and a user study further confirms its usefulness. HintDroid can not only help visually impaired individuals, but also help ordinary people understand the requirements of input components. HintDroid demo video: https://youtu.be/FWgfcctRbfI.

show abstract

Improving FlexType: Ambiguous Text Input for Users with Visual Impairments

Gaines,

Vertanen

2024

Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments

View full text Add to dashboard Cite

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

FlexType: Flexible Text Input with a Small Set of Input Gestures

Cited by 3 publications

References 33 publications

CRTypist: Simulating Touchscreen Typing Behavior via Computational Rationality

CRTypist: Simulating Touchscreen Typing Behavior via Computational Rationality

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Improving FlexType: Ambiguous Text Input for Users with Visual Impairments

Contact Info

Product

Resources

About