SpeCam

Abstract

SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it facedown) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.

Household Surface Interactions: Understanding User Input Preferences and Perceived Home Experiences

“…Different technologies have emerged for classifying individual materials to understand situation contexts. Systems such as SpecTrans [41], RadarCat [60], and SpeCam [61] have all been shown to assist in classifying materials and extending potential interactions.…”

Enabling Single-Handed Interaction in Mobile and Wearable Computing

“…In both RadarCat [9] and SpeCam [10], I started exploring new ways to use a device with simple one-handed gestures, building on the idea of placement-aware computing [2]. By using a miniature radar sensor (RadarCat) or the front-facing camera of smartphone (SpeCam), the device can then recognize the surface where it is placed or touched.…”

“…This general problem was first identified during the earliest stages of my PhD, when I was working on more expressive touch interaction with smartwatches [10,12], as I quickly realized that two hands are required. As I am now approaching the end of the third year of my PhD program, I have published several related papers, with several projects on-going and several more planned for the future, which will be described next.…”

Museum Visitor Experiences Based on Hyperspectral Image Data

“…Only little work has so far focused on using multi-or hyper spectral imaging in the field Human-Computer Interaction (HCI) to create interactive experiences or applications [11,23]. The primary reason for the limited adoption of HSI in HCI could be attributed to the high cost (which can exceed $20,000) as well as the intricate nature of these systems.…”