Integrating video games and robotic play in physical environments

Abstract

Active Learning Environments with Robotic Tangibles (ALERT) are mixed reality video gaming systems that use sensors, vision systems, and robots to provide an engaging experience that may motivate hitherto underrepresented kinds of learners to become interested in game design, programming, and careers in science, technology, engineering, and mathematics. Through the use of fiducials (i.e., meaningful markers) recognized by robots through computer vision as just-in-time instructions, users engage in spatially-based programming without the encumbrances of traditional procedural programs' syntax and structure. Since humans, robots, and video environments share many inherently spatial qualities, this natural style of physical programming is particularly well suited to fostering playful interactions with mobile robots in dynamic video environments. As these systems broaden the capabilities of video game technology and human-robot interaction (HRI) they are lowering many existing barriers to integrated videorobot game development and programming. Diverse ALERT video game scenarios and applications are enabling a broad range of gamers, learners, and developers to generate and engage in their own physically interactive games.

Toward Next Generation Mixed Reality Games: AResearch Through Design Approach

“…As we scrutinized through the state-of-the-art research, an intriguing multifaceted technodiversity was identified from current literature body. Some frequently adopted hardware settings in these MR game studies included: 1) hand-held devices such as smart phones, tablets and personal digital assistants etc., examples like the work by Samodelkin et al (2016); Cavallo and Forbes (2016); Linner et al (2005); Behmel et al (2014); Torstensson et al (2020), 2) head-mounted devices like Microsoft Hololens, Magic Leap, Oculus etc., examples like the work by Cheok et al (2002); Harley et al (2017); Kim et al (2020); Wang et al (2019); Prompolmaueng et al (2021) 3) positionfixed devices like Microsoft Kinect (Pillat et al (2012); Oswald et al (2014); Pratticò et al (2019); Jing et al (2017); Yannier et al (2013)), ceiling-/wall-mounted projectors (Kim et al (2020); Oswald et al (2014); Pratticò et al (2019); Hong et al (2017); Lupetti et al (2015); Lahey et al (2008); Hatton et al (2008); Kajastila and Hämäläinen (2014); Swearingen and Swearingen (2018)), public displays (Samodelkin et al (2016); Smith and Graham (2010)) etc. Many studies leveraged standard interactions offered by conventional I/O devices like game pads (Cheok et al (2002); Kim et al (2020); Oswald et al (2014); Pratticò et al (2019); Hong et al (2017); Khoo et al (2009); Swearingen and Swearingen (2018), touchscreens Samodelkin et al (2016);…”

“…In addition, rich alternative interactions were also found, varying greatly from one another: 1) computer vision based interaction, e.g. fiducial markers (Cheok et al (2002); Lahey et al (2008)), motion tracking (Pillat et al (2012); Jing et al (2017); Lupetti et al (2015); Kajastila and Hämäläinen (2014); Tan et al (2006)). 2) location based interaction, with players being in either an indoor or outdoor environment, including the work by Cheok et al (2002); Samodelkin et al (2016); Cavallo and Forbes (2016); Linner et al (2005); Pratticò et al (2019); Khoo et al (2009).…”

Optimists’ Creed: Brave New Cyberlearning, Evolving Utopias (Circa 2041)

“…These dreams are core to our ethical and philosophical understanding and to our appreciation of our historical heritage. Learning has evolved from ancient toys and board games, to embedded cyberlearning phenomena (Moher 2006) such as Real World Games (Waern et al 2009;Rashid et al 2006;Burak et al 2005;McGonigal 2011) and personalized environments (Lahey et al 2008;Mahoney et al 2014); the ability for individual (Papert 1980) and communal (Resnick et al 2009) appropriation of these is appreciated as being core to the reflection (Shneiderman 2003) and feedback loops that help people learn and help creative societies evolve. Open streaming content that make it real and compelling -Multi-modal modeling simulation and real world prediction provide visceral qualities that empower learners to feel experiences, as they react to immersive worlds and understand (on multiple levels) the implications.…”

TofuDraw

“…The idea of drawing the program in the physical surround rather than on a screen is inspired by Ambient Programming and others [5][6][7][8][9]. For instance, in [5] the trajectory of a robotic vehicle is programmed by placing paper cards with special markers on the floor in front of the robot where each mark maps to a behavior that the car performs as it drives over each card.…”