Humans intuitively accompany direction-giving with gestures. These gestures have been shown to have the same underlying conceptual structure as diagrams and direction-giving language, but the puzzle is how they communicate given that their form is not codified, and may in fact differ from one person or situation to the next. Based on results from a study on language and gesture in direction-giving, we describe a framework to analyze gestural images into semantic units (image description features), and to link these units to morphological features (hand shape, trajectory, etc.). This feature-based framework allows for implementing an integrated microplanner for multimodal directions that derives the form of both natural language and gesture directly from communicative goals. Using this microplanner we developed an embodied conversational agent that can perform appropriate speech and novel gestures in direction-giving conversation with real humans.

BackgroundAlzheimer’s and related dementias are on the rise, and older adults and their families are seeking accessible and effective ways to stave off or ameliorate mild cognitive impairment (MCI).AimThis pilot clinical trial (ClinicalTrials.gov Identifier: 03069391) examined neuropsychological and neurobiological outcomes of interactive physical and mental exercise.Participants and methodsOlder adults (MCI and caregivers) were enrolled in a 3-month, in-home trial of a portable neuro-exergame (the interactive Physical and Cognitive Exercise System [iPACES™]), in which they pedaled and steered along a virtual bike path to complete a list of errands (Memory Lane™). Neuropsychological function and salivary biomarkers were measured at pre-, mid-, and posttrial. Ten older adults complied with the recommended use of iPACES (complete dose; ≥2×/wk, 67% of the 15 who also had pre- and postevaluation data). Statistical analyses compared change over time and also change among those with a complete dose vs inadequate dose. Correlations between change in neuropsychological and biomarker measures were also examined.ResultsExecutive function and verbal memory increased after 3 months (p = 0.01; no significant change was found with an inadequate dose). Change in salivary biomarkers was moderately associated with increasing cognition (cortisol, r = 0.68; IGF-1, r = 0.37).ConclusionFurther research is needed, but these pilot data provide preliminary indications to suggest neuro-exergaming can impact cognitive function, perhaps via neurobiological mechanisms, and as such may provide an effective and practical way to promote healthy aging.

Dementia cases are on the rise and researchers seek innovative ways to prevent or ameliorate cognitive impairment in later life. Some research has reported that combining mental and physical exercise may benefit cognition more than either alone. This randomized pilot trial examined the feasibility and cognitive benefit for older adults (n = 30) of a single bout of neuro-exergaming (physical activity with cognitive training) using an interactive physical and cognitive exercise system (iPACES), compared with that of exergaming or neurogaming alone. Intent-to-treat and sensitivity analyses were conducted using repeated-measures ANOVA, controlling for age, sex, and education. A significant interaction effect was found for executive function (Color Trails 2), with a significant improvement in the neuro-exergaming condition. Results demonstrate feasibility for older adults to use a novel and theoretically-derived neuro-exergame, and also provide promising new evidence that neuro-exergaming can yield greater cognitive benefit than either of its component parts.

Given increasing longevity worldwide, older adults and caregivers are seeking ways to curb cognitive decline especially for those with mild cognitive impairment (MCI, now mild neurocognitive disorder, mNCD, Diagnostic and Statistical Manual of Mental Disorders, 5th ed. (DSM-V). This quasi-experimental, within-subjects pilot clinical trial was designed to replicate and extend the study of cognitive benefits for MCI by improving upon our prior interactive Physical and Cognitive Exercise Study (iPACESTM v1.0) by increasing the usability of the neuro-exergame and exploring possible underlying neurobiological mechanisms. Older adults were enrolled in a three-month, in-home trial of a portable neuro-exergame (iPACES™ v2.0) where participants pedaled and steered along a virtual bike path (Memory Lane™). Neuropsychological function was assessed at baseline after component familiarization intervals (e.g., two weeks of exercise-only, game-only, etc.) and after three months of interactive neuro-exergame intervention. Fourteen participants were enrolled in the study and seven completed the final evaluation. Intent-to-treat analyses were conducted with imputed missing data (total n = 14). Significant improvement in executive function (Stroop) was found (d = 0.68, p = 0.02) only. Changes in salivary biomarkers (cortisol and insulin-like growth factor 1; IGF-1) were significantly associated with improved cognition. Further research is needed, but pilot data suggest that a portable in-home neuro-exergame may be an additional, practical tool to fight back against cognitive decline and dementia.

We describe the first installment of the Challenge on Generating Instructions in Virtual Environments (GIVE), a new shared task for the NLG community. We motivate the design of the challenge, describe how we carried it out, and discuss the results of the system evaluation.