Neuroplex

Knowledge Infusion for Context-Aware Sensor-Based Human Activity Recognition

“…Existing NeSy methods for context-aware HAR retrieve common-sense knowledge from logic-based models (e.g., ontologies). To the best of our knowledge, three main strategies have been proposed so far to combine extracted knowledge with deep learning models: a) using knowledge to refine the deep model's output [6], b) including retrieved knowledge as additional features in the latent space [2], and c) using a loss function that penalizes predictions violating domain constraints [4,37]. However, designing and implementing knowledge models require significant human effort, and those models may not capture all the possible situations in which activities can be performed.…”

Neuro-Symbolic AI for Sensor-based Human Performance Prediction; System Architectures and Applications

“…However, little work has been proposed to tackle sensor-based predictive tasks. This can be partially attributed to the prevalence of machine learning methods, that rely on manual feature selection, over deep learning methods, that require unrealistic large amounts of data to be able to detect less frequent complex events from multiple sensor data Xing et al (2020). Moreover, the integration of prior symbolic knowledge into deep architectures remains a challenge, as in order to maintain its meaning, interpretability and logic, the reasoning process cannot be made differentiable Tiddi and Schlobach (2022); Xu et al (2018).…”

“…The hybrid system to be applied here is similar to the one in Figure 1, however the detected sequence of simple activities have a direct link to the final complex assembly activity that is detected, so for this application a hybrid system that allows for end-to-end training with only the raw sensor data and high-level assembly activity annotations given, would be ideal. Such a system, called Neuroplex, was proposed in Xing et al (2020), in which neural networks were used for perception of simple events from sensor data and neurally reconstructed reasoning models were used to detect complex events with larger spatial and temporal dependencies (after being trained with human knowledge provided as logical rules about simple-to-complex event dependencies). In addition, a semantic loss was also applied on the intermediate symbolic layer, similarly to the second approach proposed here for human state multi-class classification, to constrain the symbolic output of the neural network in order to improve the training process.…”

Neuro-Symbolic AI for Sensor-based Human Performance Prediction; System Architectures and Applications

“…However, little work has been proposed to tackle sensor-based predictive tasks. This can be partially attributed to the prevalence of machine learning methods, that rely on manual feature selection, over deep learning methods, that require unrealistic large amounts of data to be able to detect less frequent complex events from multiple sensor data Xing et al (2020). Moreover, the integration of prior symbolic knowledge into deep architectures remains a challenge, as in order to maintain its meaning, interpretability and logic, the reasoning process cannot be made differentiable Tiddi and Schlobach (2022); Xu et al (2018).…”

“…The hybrid system to be applied here is similar to the one in Figure 1, however the detected sequence of simple activities have a direct link to the final complex assembly activity that is detected, so for this application a hybrid system that allows for end-to-end training with only the raw sensor data and high-level assembly activity annotations given, would be ideal. Such a system, called Neuroplex, was proposed in Xing et al (2020), in which neural networks were used for perception of simple events from sensor data and neurally reconstructed reasoning models were used to detect complex events with larger spatial and temporal dependencies (after being trained with human knowledge provided as logical rules about simple-to-complex event dependencies). In addition, a semantic loss was also applied on the intermediate symbolic layer, similarly to the second approach proposed here for human state multi-class classification, to constrain the symbolic output of the neural network in order to improve the training process.…”