LipLearner: Customizable Silent Speech Interactions on Mobile Devices

Abstract

Figure 1: Example interaction of LipLearner. A) Voice2Lip in-situ command registration. The user records a silent speech command by vocalizing it once, then LipLearner automatically learns to lip-read it with the text recognized from the voice signal as the label. B) The command then can be used without vocalization, triggered by a silent keyword. LipLearner enables silent speech recognition which can be used in public settings (e.g., on the subway). Furthermore, it leverages incremental learning to proactively extend the model's knowledge when new samples become available.