Methods of AI for Multimodal Sensing and Action for Complex Situations

Abstract

Artificial intelligence (AI) seeks to emulate human reasoning, but is still far from achieving such results for actionable sensing in complex situations. Instead of emulating human situation understanding, machines can amplify intelligence by accessing large amounts of data, filtering unimportant information, computing relevant context, and prioritizing results (for example, answers to human queries) to provide human–machine shared context. Intelligence support can come from many contextual sources that augment data reasoning through physical, environmental, and social knowledge. We propose a decisions-to-data multimodal sensor and action through contextual agents (human or machine) that seek, combine, and make sense of relevant data. Decisions-to-data combines AI computational capabilities with human reasoning to manage data collections, perform data fusion, and assess complex situations (that is, context reasoning). Five areas of AI developments for context-based AI that cover decisions-to-data include: (1) situation modeling (data at rest), (2) measurement control (data in motion), (3) statistical algorithms (data in collect), (4) software computing (data in transit), and (5) human–machine AI (data in use). A decisions-to-data example is presented of a command-guided swarm requiring contextual data analysis, systems-level design, and user interaction for effective and efficient multimodal sensing and action.

On the current state of combining human and artificial intelligence for strategic organizational decision making

“…Researchers of this category agree on the stages of input-process-output, with related definitions on data being at rest, in collection, in transition, in motion, or in use. Parallel to this, respective applications increase in ability from purely statistical Baryannis et al (2019a) and Blasch et al (2019) Special applications provided by the sample Business Research AI, which some researchers do not even classify as AI (Baryannis et al 2019a), to human-machine AI (Blasch et al 2019), supporting the framework in Sect. 2.1.2.…”

“…Scholars argue that determining which application to use depends on the type, quantity, and quality of data available, resulting in various necessities to handle the data, such as classification, clustering, or detection of connections (see Table 3). Moreover, as several applications can be used for both top-down and bottom-up approaches (Flath and Stein 2018;Mühlroth and Grottke 2018;Baryannis et al 2019a, b;Blasch et al 2019), the purpose for which the specific application should be used is identified as an additional influence (Blasch et al 2019).…”

“…Articles use and recommend a hybrid approach, as mathematical models are, on the one hand, found to be less capable of handling a large amount of data, which, on the other hand, is needed to train machine learning applications, often based on mathematical ones (Baryannis et al 2019b;Blasch et al 2019). Therefore, Simon's (1995) definition of AI applications being more than mathematical theorems is supported.…”

“…Information collection is seen as a task to be completely fulfilled by AI. It is related to the generation of information from various and numerous sources with differing techniques, such as natural language processing, text mining, or other data mining possibilities (Baryannis et al Blasch et al 2019). Nevertheless, executing feature engineering afterwards to reduce input bias as far as possible is said to be necessarily done by humans, assisted by top-down applications (Flath and Stein 2018).…”

“…Relating to Fig. 2, most applications that the articles refer to can be clustered as top-down, as they are not able to act self-consciously (i.e., human-like; Blasch et al 2019). With increasing research and development efforts, however, the literature expects the capabilities of AI applications to increase and to shift to the right (Mühlroth and Grottke 2018; Colombo 2019).…”