Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges

Qiu, Sen; Zhao, Hongkai; Jiang, Nan; Wang, Zhelong; Liu, Long; An, Yi; Zhao, Hongyu; Miao, Xin; Liu, Ruichen; Fortino, Giancarlo

doi:10.1016/j.inffus.2021.11.006

Cited by 410 publications

(155 citation statements)

References 225 publications

Supporting

Mentioning

152

Contrasting

Unclassified

Order By: Relevance

“…Although it has recently been demonstrated that using multisensor data fusion has higher recognition accuracy for complex human movements than a single sensor [ 15 ], our study aimed to examine the trajectory of sway during quiet stance, which had no movement complexity. Therefore, we positioned a single sensor in the middle of the user's waist, at the L5 vertebra in the lumbar area of the trunk, as evidenced by a previous study that indicated that the central sensors had a higher recognition rate than the left and right ones [ 23 ].…”

Section: Discussionmentioning

confidence: 99%

Application of Machine Learning to Predict Trajectory of the Center of Pressure (COP) Path of Postural Sway Using a Triaxial Inertial Sensor

Wantanajittikul

Wiboonsuntharangkoon

Chuatrakoon

et al. 2022

The Scientific World Journal

View full text Add to dashboard Cite

Postural sway indicates controlling stability in response to standing balance perturbations and determines risk of falling. In order to assess balance and postural sway, costly laboratory equipment is required, making it impractical for clinical settings. The study aimed to develop a triaxial inertial sensor and apply machine learning (ML) algorithms for predicting trajectory of the center of pressure (COP) path of postural sway. Fifty-three healthy adults, with a mean age of 46 years, participated. The inertial sensor prototype was investigated for its concurrent validity relative to the COP path length obtained from the force platform measurement. Then, ML was applied to predict the COP path by using sensor-sway metrics as the input. The results of the study revealed that all variables from the sensor prototype demonstrated high concurrent validity against the COP path from the force platform measurement (ρ > 0.75; p < 0.001 ). The agreement between sway metrics, derived from the sensor and ML algorithms, illustrated good to excellent agreement (ICC; 0.89–0.95) between COP paths from the sensor metrics, with respect to the force plate measurement. This study demonstrated that the inertial sensor, in comparison to the standard tool, would be an option for balance assessment since it is of low-cost, conveniently portable, and comparable to the accuracy of standard force platform measurement.

show abstract

Section: Discussionmentioning

confidence: 99%

Application of Machine Learning to Predict Trajectory of the Center of Pressure (COP) Path of Postural Sway Using a Triaxial Inertial Sensor

Wantanajittikul

Wiboonsuntharangkoon

Chuatrakoon

et al. 2022

The Scientific World Journal

View full text Add to dashboard Cite

show abstract

“…A key issue that the community is paying increasing attention to is model robustness and reliability [308,309]. One common way to improve robustness is to leverage the benefits of multiple types of sensors together to create multi-sensory systems [249,[310][311][312][313][314].…”

Section: Model Robustnessmentioning

confidence: 99%

Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances

Zhang

et al. 2022

Sensors

244

View full text Add to dashboard Cite

Mobile and wearable devices have enabled numerous applications, including activity tracking, wellness monitoring, and human–computer interaction, that measure and improve our daily lives. Many of these applications are made possible by leveraging the rich collection of low-power sensors found in many mobile and wearable devices to perform human activity recognition (HAR). Recently, deep learning has greatly pushed the boundaries of HAR on mobile and wearable devices. This paper systematically categorizes and summarizes existing work that introduces deep learning methods for wearables-based HAR and provides a comprehensive analysis of the current advancements, developing trends, and major challenges. We also present cutting-edge frontiers and future directions for deep learning-based HAR.

show abstract

“…Dedicated pre-processing pipelines will be applied for different sensor types, but fusion across sensors will provide a spatially and temporally contextualised overview of ADL behaviours and anomalies. We will explore which sensors are best combined for different activities and at what level of integration, and also the optimal stages at which to fuse activity with location data, ranging from early (naïve multimodal feature set integration) to late (hierarchical integration of posterior probabilities from deep learning models trained on individual feature sets) fusion strategies [23,56].…”

Section: Location and Activity Detection From Sphere Sensor Datamentioning

confidence: 99%

“…This sequential dependence of states can be modelled using Viterbi decoding, which tracks a maximum-likelihood path through a series of noisy, probabilistic transitions in a Hidden Markov Model (HMM) [56,57]. We will compare HMM activity classification and localization performance to deep learning architectures capable of leveraging spatial and temporal relationships in the multimodal feature space such as convolutional neural networks and long short-term memory networks [23]. We will compare these with pipelines exploiting both activity and location data together for further disambiguation of behaviour.…”

Section: Location and Activity Detection From Sphere Sensor Datamentioning

confidence: 99%

“…These ecologically valid, everyday behaviours can be measured continuously and unobtrusively in the home, using networked fixed sensors, or portable sensors in wearables, smartphones and tablets [16]. Fusion across sensing modalities permits fine-grained contextual analysis of how engagement in different ADL covaries over days, weeks and months, and automated machine learning (ML) techniques can leverage intraindividual variability reflective of early cognitive changes [15,23]. These technologies have potential to help clinicians build personalised and holistic digital phenotypes for prediction of neurodegeneration [17,24].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The use of home-based conversations for detecting early dementia: Protocol for the CUBOId TV task

Kumpik

Santos-Rodríguez

Selwood

et al. 2022

Preprint

View full text Add to dashboard Cite

IntroductionLimitations in effective dementia therapies mean that early diagnosis and monitoring are critical for disease management, but current clinical tools are impractical and/or unreliable, and disregard short-term symptom variability. Behavioural biomarkers of cognitive decline, such as speech, sleep and activity patterns, can manifest prodromal pathological changes. They can be continuously measured at home with smart sensing technologies, and permit leveraging of interpersonal interactions for optimising diagnostic and prognostic performance. Here we describe the ContinUous behavioural Biomarkers Of cognitive Impairment (CUBOId) study, which explores the feasibility of multimodal data fusion for in-home monitoring of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). The report focuses on a subset of CUBOId participants who perform a novel speech task, the “TV task”, designed to track changes in ecologically valid conversations with disease progression.Methods and AnalysisParticipants have diagnoses of MCI or AD, and controls are their live-in partners with no such diagnosis. Multimodal activity data were passively acquired from wearables and in-home fixed sensors over periods ranging from 8 to 25 months. At 2 time points participants completed the TV task over 5 days by recording audio of their conversations as they watched a favourite TV programme, with further testing to be completed after removal of the sensor installations. Behavioural testing is supported by neuropsychological assessment for deriving behavioural ground truths on cognitive status. Deep learning will be used to generate fused multimodal activity-speech embeddings for optimisation of diagnostic and predictive performance from speech alone.Ethics and disseminationCUBOId was approved by an NHS Research Ethics Committee (Wales REC; ref: 18/WA/0158) and is sponsored by University of Bristol. It is also supported by the National Institute for Health Research (NIHR) Clinical Research Network West of England. Results will be reported at conferences and in peer-reviewed scientific journals.ARTICLE SUMMARYStrengths and limitationsTo our knowledge, this is the first study to simultaneously characterise longitudinal, ecologically valid diagnostic trajectories across a broad range of behavioural domains relevant to cognitive decline “in the wild”, allowing construction of shared multimodal embeddings for dementia diagnosis from speech aloneParticipants’ live-in partners are contextually matched controls, accounting for differences across participants’ home environments and lifestylesMeasurements of cognitive status from neuropsychological testing form ground truths for behavioural biomarkers of cognitive declineLimited sample size and restricted demographics may confer limited generalisability of findings to other populationsBehavioural and neuropsychological testing timelines disrupted and desynchronised due to COVID-19

show abstract

Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges

Cited by 410 publications

References 225 publications

Application of Machine Learning to Predict Trajectory of the Center of Pressure (COP) Path of Postural Sway Using a Triaxial Inertial Sensor

Application of Machine Learning to Predict Trajectory of the Center of Pressure (COP) Path of Postural Sway Using a Triaxial Inertial Sensor

Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances

The use of home-based conversations for detecting early dementia: Protocol for the CUBOId TV task

Contact Info

Product

Resources

About