Relevance in cyber‐physical systems with humans in the loop

Abstract

SUMMARYIn cyber-physical systems such as intelligent lighting, the system responds autonomously to observed changes in the environment. In such systems, more than one output may be acceptable for a given input scenario. This type of relationship between the input and output makes it difficult to analyze machine learning algorithms using commonly used performance metrics such as classification accuracy (CA). CA only measures whether a predicted output is right or not, whereas it is more important to determine whether the predicted output is relevant for the given context or not. In this direction, we introduce a new metric, the relevance score (RS) that is effective for the class of applications where user perception leads to nondeterministic input-output relationships. RS determines the extent by which a predicted output is relevant to the user's context and behaviors, taking into account the variability and bias that come with human perception factors. We assess the performance of a number of machine learning algorithms, using different datasets, including data from an intelligent lighting pilot. We find that using RS instead of CA is appropriate to analyze the performance of conventional machine learning algorithms, particularly for the class of non-deterministic multiple-output problems. Our method may be applied to other scenarios in which cyber-physical systems involve humans in the control loop.

Towards Internet of Underground Things in smart lighting: A statistical model of wireless underground channel

“…Smart lighting is an emerging field [8], [9], and there exist only few architectures. Use of the IOUT communications in this field has not been investigated before.…”

“…A potential application of the IOUT is in the area of smart lighting [9], [8], [22], where cables are buried underground for control of intelligent lighting systems. The overview of the smart lighting IOUT system architecture is shown in Fig.…”

Runtime evaluation of cognitive systems for non-deterministic multiple output classification problems

“…RS is the degree of relevance of a predicted output for a given context with respect to the actual output [18,19]. When there is a mismatch between predicted and actual outputs, RS will account for the level of inconsistency of non-deterministic multiple-output cases.…”

Co-Simulation of Cyber Physical Systems with HMI for Human in the Loop Investigation

“…A comprehensive survey of the current state of human in the loop CPS efforts has been performed in (Nunes et al 2015). In (Gopalakrishna et al 2017), dealing with machine-learning for human in the loop CPS, a new way of determining the accuracy of an output based on a relevance score taking into account the variability and bios of the human perception. Similar for most research in Human CPS (Gopalakrishna et al 2017, Nunes et al 2015, Lieber and Fass 2011, is the acknowledgement of multidisciplinary technical challenges and argument that the traditional development process, where human impact is not an integrated part of the process, will not be sufficient for future products.…”