Figure 1. CommandBoard creates a new command gesture input space above a soft keyboard. Users can: a) type 'happy' and use a dynamic guide to style it as bold; b) type 'brightn', draw an execute gesture and adjust the brightness slider; c) type 'sans', choose 'sans mono' and draw an execute gesture to change the font; d) type 'color', select yellow in the marking menu to change the brush color.

Gesture-typing is an efficient, easy-to-learn, and errortolerant technique for entering text on software keyboards. Our goal is to "recycle" users' otherwise-unused gesture variation to create rich output under the users' control, without sacrificing accuracy. Experiment 1 reveals a high level of existing gesture variation, even for accurate text, and shows that users can consciously vary their gestures under different conditions. We designed an Expressive Keyboard for a smart phone which maps input gesture features identified in Experiment 1 to a continuous output parameter space, i.e. RGB color. Experiment 2 shows that users can consciously modify their gestures, while retaining accuracy, to generate specific colors as they gesture-type. Users are more successful when they focus on output characteristics (such as red) rather than input characteristics (such as curviness). We designed an app with a dynamic font engine that continuously interpolates between several typefaces, as well as controlling weight and random variation. Experiment 3 shows that, in the context of a more ecologically-valid conversation task, users enjoy generating multiple forms of rich output. We conclude with suggestions for how the Expressive Keyboard approach can enhance a wide variety of gesture recognition applications.

Figure 1: (a) Two users exchanging a mediated handshake with SansTouch at 2-meter distance. (b) To trigger the handshake stimuli, both users synchronously mimic the hand movement of a handshake as in real life. (c) Each user wears the multimodal hand device while holding a smartphone. .