PixelTone

Abstract

is is a shirt" "Change the color of the shirt" a b c Figure 1. With PIXELTONE, users speak to edit their images instead of hunting through menus. a) The user selects the person's shirt and says "This is a shirt." PIXELTONE associates the tag "shirt" with the selected region. b) The user tells PIXELTONE to "Change the color of the shirt," and c) PIXELTONE applies a hue adjustment to the image and offers a slider so that the user can explore different colors. ABSTRACTPhoto editing can be a challenging task, and it becomes even more difficult on the small, portable screens of mobile devices that are now frequently used to capture and edit images. To address this problem we present PIXELTONE, a multimodal photo editing interface that combines speech and direct manipulation. We observe existing image editing practices and derive a set of principles that guide our design. In particular, we use natural language for expressing desired changes to an image, and sketching to localize these changes to specific regions. To support the language commonly used in photoediting we develop a customized natural language interpreter that maps user phrases to specific image processing operations. Finally, we perform a user study that evaluates and demonstrates the effectiveness of our interface.

Inclusive Voice Interaction Techniques for Creative Object Positioning

“…Exploring alternative multimodal approaches to support creative work is also a potentially fruitful arearesearch has examined the use of speech combined with gestures [10,17] and touch input [26,32], although other methods could also be beneficial (e.g. using gaze for selecting locations within a canvas and speech to confirm movements, thus helping address known issues such as the Midas Touch [19]).…”

“…Other studies have also explored multimodal approaches for creating artistic work, image editing, and graphical manipulationfor instance, Laput et al [26] presented the PIXELTONE application where direct manipulation (via touch) is used to select parts of the image, along with a limited set of high level voice commands to perform image editing operations (e.g. applying filters).…”

Scones

“…Closely related to our domain, DrawAFriend uses crowdsourced data through a Game With a Purpose (GWAP) [31] to refne and correct users' sketch strokes [19]. PixelTone additionally uses natural language speech input to apply flters to annotated images [17]. Most relevant to Scones, Ribeiro and Igarashi introduced a two-way sketch-based communication game for users to iteratively edit sketches using a direct manipulation interface [24].…”

Crosspower

“…For example, VoiceCuts enabled users to issue short, partial, and shortcut-like commands to quickly perform intended actions in creativity-based applications [30]. PixelTone enabled users to express desired operations with natural language in an image editing application [32]. DataTone [18] and Orko [46] also enabled users to issue natural language commands to explore data visualizations.…”