Semantic Entity-Component State Management Techniques to Enhance Software Quality for Multimodal VR-Systems

Abstract

Modularity, modifiability, reusability, and API usability are important software qualities that determine the maintainability of software architectures. Virtual, Augmented, and Mixed Reality (VR, AR, MR) systems, modern computer games, as well as interactive human-robot systems often include various dedicated input-, output-, and processing subsystems. These subsystems collectively maintain a real-time simulation of a coherent application state. The resulting interdependencies between individual state representations, mutual state access, overall synchronization, and flow of control implies a conceptual close coupling whereas software quality asks for a decoupling to develop maintainable solutions. This article presents five semantics-based software techniques that address this contradiction: Semantic grounding, code from semantics, grounded actions, semantic queries, and decoupling by semantics. These techniques are applied to extend the well-established entity-component-system (ECS) pattern to overcome some of this pattern's deficits with respect to the implied state access. A walk-through of central implementation aspects of a multimodal (speech and gesture) VR-interface is used to highlight the techniques' benefits. This use-case is chosen as a prototypical example of complex architectures with multiple interacting subsystems found in many VR, AR and MR architectures. Finally, implementation hints are given, lessons learned regarding maintainability pointed-out, and performance implications discussed.

Context‐Aware Mixed Reality: A Learning‐Based Framework for Semantic‐Level Interaction

“…Prior approaches have studied context and semantic understandings in 3D virtual environments, for example semantic inferring in interactive visual data exploration [NEF12]; enhancing software quality for multi‐modal virtual reality (VR) systems [FWL17]; visual text analytics [EFN12]; and interactive urban visualization [DZMQ16]. Context awareness is also introduced in computer‐aided graphic design, such as inbetweening of animation[Yan18]; 3D particle clouds selection[YEII16]; and illustrative volume rendering [RBG07].…”

Productivity Metric Estimation: Comprehensive Examine of Efficiency, Effectiveness and Value Based Metrics

“…Test phase is negative and its success depends greatly on the bugs determined to improve quality of code segment. Trendowiczet al [24]discussed the factors that are going to influence quality of code segment. Quality of code segment greatly depends upon metric evaluation mechanisms and SRS satisfaction criteria's.…”

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

“…Hence, they are less suited to perform semantic integration. On the contrary, platforms with an integrated MMI framework, like SGIM and virtuelle Werkstatt [56], ICare [57], and Simulator X [20], commonly provide a uniform access scheme to the application's context.…”

“…On the other hand, software quality asks for a decoupling to develop maintainable solutions. This contradiction negatively effects the availability of these frameworks [20,58]. This challenge is exacerbated by the integration of subsystems that reflect the overall state and behavior, like a semantic fusion subsystem which performs semantic integration [22,59,60].…”

“…This analysis has been given different names, ranging from combining [15] to multimodal integration [16]. More widely the term fusion is used, e.g., multimodal input fusion [17] or the more recently used multimodal fusion [18][19][20]. Multimodal fusion is applied for a wide variety of scenarios.…”