Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results

Speth, Jeremy; Vance, Nathan; Czajka, Adam; Bowyer, Kevin W.; Wright, Diane; Flynn, Patrick J.

doi:10.1109/ijcb52358.2021.9484409

Cited by 24 publications

(9 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used the generated bounding box coordinates to crop ROIs from each frame for each body part and downsized these ROIs to 64x64 pixels using bicubic interpolation. The RPNet model we used was trained on the DDPM dataset [37,42] where the frame rate is 90 frames per second (fps), which is the same as our MSPM dataset. The trained RPNet model was fed video clips of 135 frames (1.5 seconds) as described in the original paper [36].…”

Section: Learning-based Methodsmentioning

confidence: 99%

“…Most approaches leverage contrastive training strategies, where similar pairs of samples are pulled closer and different samples are repelled [10,40,43,48]. The first non-contrastive approach leverages strong periodic priors to encourage the model to predict sparse signals in the frequency domain [38].…”

Section: Related Workmentioning

confidence: 99%

“…These three methods perform well for both stationary and moving subjects, which are characteristic of our MSPM dataset. The RPNet model was trained on face videos from the large and challenging deception detection and physiological monitoring dataset (DDPM) [37,42]. The model was not fine-tuned on any data from MSPM, so we tested how well the model could transfer to new subjects, lighting, standoff, and movement.…”

Section: Multi-site Rppg Approachmentioning

confidence: 99%

“…UBFC-rPPG [5] contains occasional movements and subjects under stress from mathematical games. DDPM [37,42] was the first long-form dataset containing unconstrained facial movements. Synthetic face rPPG datasets containing avatars also only contain faces [24,46].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Full-Body Cardiovascular Sensing with Remote Photoplethysmography

Lü¹,

Speth²,

Vance³

et al. 2023

Preprint

View full text Add to dashboard Cite

Remote photoplethysmography (rPPG) allows for noncontact monitoring of blood volume changes from a camera by detecting minor fluctuations in reflected light. Prior applications of rPPG focused on face videos. In this paper we explored the feasibility of rPPG from non-face body regions such as the arms, legs, and hands. We collected a new dataset titled Multi-Site Physiological Monitoring (MSPM), which will be released with this paper. The dataset consists of 90 frames per second video of exposed arms, legs, and face, along with 10 synchronized PPG recordings. We performed baseline heart rate estimation experiments from non-face regions with several state-of-the-art rPPG approaches, including chrominance-based (CHROM), planeorthogonal-to-skin (POS) and RemotePulseNet (RPNet). To our knowledge, this is the first evaluation of the fidelity of rPPG signals simultaneously obtained from multiple regions of a human body. Our experiments showed that skin pixels from arms, legs, and hands are all potential sources of the blood volume pulse. The best-performing approach, POS, achieved a mean absolute error peaking at 7.11 beats per minute from non-facial body parts compared to 1.38 beats per minute from the face. Additionally, we performed experiments on pulse transit time (PTT) from both the contact PPG and rPPG signals. We found that remote PTT is possible with moderately high frame rate video when distal locations on the body are visible. These findings and the supporting dataset should facilitate new research on nonface rPPG and monitoring blood flow dynamics over the whole body with a camera.

show abstract

Section: Learning-based Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Multi-site Rppg Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Full-Body Cardiovascular Sensing with Remote Photoplethysmography

Lü¹,

Speth²,

Vance³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The five studies that used RBF SVM [70,77,89,109,112] measured their performances by accuracy, which ranges from 0.5650 to 0.8247, with mean at 0.6808 ± 0.1101. For more details, see section 5.3.2 in S6 File.…”

Section: Artificial Neuralmentioning

confidence: 99%

Deception detection with machine learning: A systematic review and statistical analysis

et al. 2023

View full text Add to dashboard Cite

Several studies applying Machine Learning to deception detection have been published in the last decade. A rich and complex set of settings, approaches, theories, and results is now available. Therefore, one may find it difficult to identify trends, successful paths, gaps, and opportunities for contribution. The present literature review aims to provide the state of research regarding deception detection with Machine Learning. We followed the PRISMA protocol and retrieved 648 articles from ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. 540 of them were screened (108 were duplicates). A final corpus of 81 documents has been summarized as mind maps. Metadata was extracted and has been encoded as Python dictionaries to support a statistical analysis scripted in Python programming language, and available as a collection of Jupyter Lab Notebooks in a GitHub repository. All are available as Jupyter Lab Notebooks. Neural Networks, Support Vector Machines, Random Forest, Decision Tree and K-nearest Neighbor are the five most explored techniques. The studies report a detection performance ranging from 51% to 100%, with 19 works reaching accuracy rate above 0.9. Monomodal, Bimodal, and Multimodal approaches were exploited and achieved various accuracy levels for detection. Bimodal and Multimodal approaches have become a trend over Monomodal ones, although there are high-performance examples of the latter. Studies that exploit language and linguistic features, 75% are dedicated to English. The findings include observations of the following: language and culture, emotional features, psychological traits, cognitive load, facial cues, complexity, performance, and Machine Learning topics. We also present a dataset benchmark. Main conclusions are that labeled datasets from real-life data are scarce. Also, there is still room for new approaches for deception detection with Machine Learning, especially if focused on languages and cultures other than English-based. Further research would greatly contribute by providing new labeled and multimodal datasets for deception detection, both for English and other languages.

show abstract