Fine-grained activities recognition with coarse-grained labeled multi-modal data

Hu, Zhizhang; Yu, Tong; Zhang, Yue; Pan, Shijia

doi:10.1145/3410530.3414320

Cited by 13 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditional coarse-grained activities are concerned with scene-level information, usually involving discrete activities with highly distinct inter-class features. They do not include detailed features on continuous activities in applications [11]. These highly distinct inter-class features are presented in many popular benchmark activity datasets, such as KTH [12], UT-interaction [13], and UTKinect-Action3D [14].…”

Section: Related Workmentioning

confidence: 99%

Fine-grained activity classification in assembly based on multi-visual modalities

Chen¹,

Zendehdel²,

Leu³

et al. 2023

J Intell Manuf

View full text Add to dashboard Cite

Assembly activity recognition and prediction help to improve productivity, quality control, and safety measures in smart factories. This study aims to sense, recognize, and predict a worker's continuous fine-grained assembly activities in a manufacturing platform. We propose a two-stage network for workers' fine-grained activity classification by leveraging scene-level and temporal-level activity features. The first stage is a feature awareness block that extracts scene-level features from multi-visual modalities, including red-green-blue (RGB) and hand skeleton frames. We use the transfer learning method in the first stage and compare three different pre-trained feature extraction models. Then, we transmit the feature information from the first stage to the second stage to learn the temporal-level features of activities. The second stage consists of the Recurrent Neural Network (RNN) layers and a final classifier. We compare the performance of two different RNNs in the second stage, including the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU). The partial video observation method is used in the prediction of fine-grained activities. In the experiments using the trimmed activity videos, our model achieves an accuracy of > 99% on our dataset and > 98% on the public dataset UCF 101, outperforming the state-of-the-art models. The prediction model achieves an accuracy of > 97% in predicting activity labels using 50% of the onset activity video information. In the experiments using an untrimmed video with continuous assembly activities, we combine our recognition and prediction models and achieve an accuracy of > 91% in real time, surpassing the state-of-the-art models for the recognition of continuous assembly activities.

show abstract

Section: Related Workmentioning

confidence: 99%

Fine-grained activity classification in assembly based on multi-visual modalities

Chen¹,

Zendehdel²,

Leu³

et al. 2023

J Intell Manuf

View full text Add to dashboard Cite

show abstract

“…Physical vibration signals induced by people in the buildings are used to indirectly infer human information for both physical and physiology information, include and not limited to identity (Pan et al, 2017 ), location (Mirshekari et al, 2018 ; Drira et al, 2021 ), activity (Hu et al, 2020 ; Sun et al, 2020 ), heart beat (Jia et al, 2016 ), and gait (Fagert et al, 2019 ). The intuition is that people induce physical vibrations all the time, such as stepping on the floor, heart-pounding in the chest, etc.…”

Section: Related Workmentioning

confidence: 99%

“…System often require sensors to have overlapping sensing area to enable applications such as step-level localization (Mirshekari et al, 2018 ), gait analysis (Fagert et al, 2020 ), and activity recognition (Hu et al, 2020 ; Sun et al, 2020 ). On the other hands, for applications such as localization and gait analysis, sensor devices' locations in the room coordinates are also needed.…”

Section: Related Workmentioning

confidence: 99%

AutoLoc: Autonomous Sensor Location Configuration via Cross Modal Sensing

Rohal

Zhang

Ruiz³

et al. 2022

Front. Big Data

Self Cite

View full text Add to dashboard Cite

Internet-of-Things (IoT) systems have become pervasive for smart homes. In recent years, many of these IoT sensing systems are developed to enable in-home long-term monitoring applications, such as personalized services in smart homes, elderly/patient monitoring, etc. However, these systems often require complicated and expensive installation processes, which are some of the main concerns affecting users' adoption of smart home systems. In this work, we focus on floor vibration-based occupant monitoring systems, which enables non-intrusive in-home continuous occupant monitoring, such as patient step tracking and gait analysis. However, to enable these applications, the system would require known locations of vibration sensors placed in the environment. Current practice relies on manually input of location, which makes the installation labor-intensive, time consuming, and expensive. On the other hand, without known location of vibration sensors, the output of the system does not have intuitive physical meaning and is incomprehensive to users, which limits the systems' usability. We present AutoLoc, a scheme to estimate the location of the vibration sensors in a two-dimensional space in the view of a nearby camera, which has spatial physical meaning. AutoLoc utilizes occupants' walking events captured by both vibration sensors and the co-located camera to estimate the vibration sensors' location in the camera view. First, AutoLoc detects and localizes the occupant's footsteps in the vision data. Then, it associates the time and location of the event to the floor vibration data. Next, the extracted vibration data of the given event from multiple vibration sensors are used to estimate the sensors' locations in the camera view coordinates. We conducted real-world experiments and achieved up to 0.07 meters localization accuracy.

show abstract

“…the occupant action. As a result, the captured vibration can be used to infer the building occupant information, such as presence [42], occupancy [43], identity [44], activity [30], and location [39]. The advantages of this sensing modality include NLOS sensing without much privacy concerns.…”

Section: Vibration-based Occupancy Sensingmentioning

confidence: 99%

Modes

Rajabi

Ding

et al. 2022

Proceedings of the Thirteenth ACM International Conference on Future Energy Systems

Self Cite

View full text Add to dashboard Cite

Buildings account for more than 40% of the energy US primary energy consumption. Of all the building services, heating, ventilation, and air-conditioning (HVAC) account for almost 50% of that energy use. Despite all the resources used, many users are not satisfied with the comfort conditions in buildings. The main problems for this lack of balance between energy use and quality of comfort are the lack of occupancy information, real user comfort feedback, and easily built zone thermodynamic models available to the Building Management Systems (BMS). In our work, we focus on occupancy sensing. While occupancy sensing is very important and there are multiple different sensing technologies used to address this issue, a precise and reliable measurement of occupancy remains elusive.In this paper, we propose MODES, a Multi-sensor Occupancy Data-driven Estimation System for Smart Buildings. Leveraging on two different state-of-the-art sensing techniques available in the literature (vibration and thermal sensors), both being capable of counting the number of occupants in any particular zone. The two occupancy estimations are then fused using a data-driven optimization process for sensor fusion to create an improved estimate. This newly updated estimate is further used together with a data-driven occupancy model as input of a particle filter to provide an even more accurate estimate. We tested the system in a commercial building under realistic conditions using real experimental occupancy data traces with users doing their daily routines. We showed that MODES can improve occupancy estimation by 40% from vibration sensors, 19% from thermal sensors, and 30% from state-of-the-art sensor fusion schemes. Moreover, we show that this is possible with minimum data training requirements, needing 7 days of training data to train the fusion system. We also run several EnergyPlus simulations using an occupancy-driven HVAC controller under different occupancy errors to show the impact that more accurate occupancy sensing schemes can have on the overall energy usage and quality of comfort and air ventilation. Our study shows that MODES can save up to 77% of energy use in a building while improving the quality of comfort by 10%. 1

show abstract

Fine-grained activities recognition with coarse-grained labeled multi-modal data

Cited by 13 publications

References 23 publications

Fine-grained activity classification in assembly based on multi-visual modalities

Fine-grained activity classification in assembly based on multi-visual modalities

AutoLoc: Autonomous Sensor Location Configuration via Cross Modal Sensing

Modes

Contact Info

Product

Resources

About