Fluid interaction with high-resolution wall-size displays

Abstract

As computers become more ubiquitous, direct interaction with wall-size, high-resolution displays will become commonplace. The familiar desktop computer interface is ill-suited to the affordances of these screens, such as size, and capacity for using pen or finger as primary input device.Current Graphical User Interfaces (GUIs) do not take into account the cost of reaching for a faraway menu bar, for example, and they rely heavily on the keyboard for rapid interactions. GUIs are extremely powerful, but their interaction style contrasts sharply with the casual interaction style available with traditional wall-size displays such as whiteboards and bulletin boards.This thesis explores how to bridge the gap between the power provided by current desktop computer interfaces and the fluid use of whiteboards and pin-boards. Based on our observations of fluid expert interactions from everyday life, such as driving a car or playing a violin, we have designed and built a fluid interaction framework which encourages gesture memory, reduces the need for dialog with the user, and provides a scoping mechanism for modes. Together, these features progressively make the cognitive load of using the interface disappear. The user becomes free to focus on other tasks, the same way one can drive a car while conversing with a passenger.To validate our design, we built the Stanford Interactive Mural, a 9 Mpixel whiteboard-size screen, evaluated the performance of our proposed menu system FlowMenu, and implemented two applications using our framework. The Geometer's Workbench allows one to explore differential geometry; PostBrainstorm is a brainstorm tool that lets users gather and organize sketches, snapshots of physical documents, and a variety of digital documents on the Interactive Mural. PostBrainstorm was tested in brainstorming sessions by professional designers. It demonstrates the feasibility of fluid, transparent interactions for complex, real life applications. v Acknowledgments I would like to thank the many people who helped me as I did the work described in this dissertation. This work would not have been possible without their support.It was a real privilege to work with my advisor, Terry Winograd. We had many long and insightful discussions exploring how human computer interactions can be improved. Even though we sometimes disagreed on the specifics, I believe we share the same vision in which humans and computers will be engaged in a synergetic relationship, each leveraging the strength of the other while minimizing the other's weaknesses. Terry was always ready to discuss new ideas and was always very supportive during the four or so years of our collaboration.

AppLens and launchTile

“…Gestures have proven a popular interaction alternative when hardware alone fails to effectively support user tasks, typical of many nontraditional devices, from mobile computers to wall-sized displays [9]. Gestures can be very efficient, combining both command and operand in a single motion, and are space-conserving, reducing the need for software buttons and menus.…”

CrossY

“…In the recent years, several systems also challenged the use of the point-and-click interface for pen computing in whiteboard environments such as Tivoli [22], FlatLand [20] and PostBrainstorm [10], on the desktop [26], or for pen computing [27]. These systems are in general tuned to a certain class of applications (such as brainstorming for example) and did not focus on crossing as the sole interaction paradigm.…”

Multimodal user interfaces for a travel assistant

“…Les interfaces classiques requièrent généralement une décomposition des actions en sous-actions nécessitant des interactions différentes (par exemple lorsque l'utilisateur veut "zoomer" l'interface il lui faut généralement d'abord effectuer la sous-action "se mettre en mode zoom" en cliquant sur un bouton, puis la sous-action "zoomer" en manipulant la souris, un potentiomètre, etc.). Cette décomposition semble avoir un impact non négligeable sur la fluidité de l'interaction comme en attestent les travaux de Buxton [4] et d'autres études (en particulier sur les Flow menus, une technique d'interaction proche des Control menus et légèrement postérieure [7]). Enfin, de même que les Pie et Marking menus dont ils s'inspirent [5,6], grâce à leur disposition spatiale les Control menus permettent aux utilisateurs de se remémorer les gestes servant à sélectionner les actions.…”