Globem

Abstract

There is a growing body of research revealing that longitudinal passive sensing data from smartphones and wearable devices can capture daily behavior signals for human behavior modeling, such as depression detection. Most prior studies build and evaluate machine learning models using data collected from a single population. However, to ensure that a behavior model can work for a larger group of users, its generalizability needs to be verified on multiple datasets from different populations. We present the first work evaluating cross-dataset generalizability of longitudinal behavior models, using depression detection as an application. We collect multiple longitudinal passive mobile sensing datasets with over 500 users from two institutes over a two-year span, leading to four institute-year datasets. Using the datasets, we closely re-implement and evaluated nine prior depression detection algorithms. Our experiment reveals the lack of model generalizability of these methods. We also implement eight recently popular domain generalization algorithms from the machine learning community. Our results indicate that these methods also do not generalize well on our datasets, with barely any advantage over the naive baseline of guessing the majority. We then present two new algorithms with better generalizability. Our new algorithm, Reorder, significantly and consistently outperforms existing methods on most cross-dataset generalization setups. However, the overall advantage is incremental and still has great room for improvement. Our analysis reveals that the individual differences (both within and between populations) may play the most important role in the cross-dataset generalization challenge. Finally, we provide an open-source benchmark platform GLOBEM- short for Generalization of Longitudinal BEhavior Modeling - to consolidate all 19 algorithms. GLOBEM can support researchers in using, developing, and evaluating different longitudinal behavior modeling methods. We call for researchers' attention to model generalizability evaluation for future longitudinal human behavior modeling studies.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data

“…Table 2 summarizes the data used for analysis. 3900 samples were analyzed from 650 individuals, a large cohort and sample size compared to most studies to date analyzing associations between sensed-behaviors and mental health 4 , 5 , 25 , 35 , 36 . A sample was a set of sensed-behaviors, summarized over 2 weeks, corresponding to the average PHQ-8 response collected during a single weekly reporting period.…”

“…Initial work showed that depression risk could be predicted from sensed-behavioral data at a similar accuracy to general practitioners 10 in small populations 5 , 11 . More recent work shows that these AI tools predict depression risk at an accuracy only slightly better than a coin flip in larger, more diverse samples 4 , 6 , 12 , 13 . This prior work has not specifically explored why accuracy is reduced in larger samples, and it is unclear how to improve AI tools for clinical use.…”

Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Intervention

“…3.1.1 Feature Design. We design a set of five passive sensing feature categories [59,87,89] to capture smartphone overuse behavior: (a) Phone and App Usage. Understanding smartphone overuse requires a thorough analysis of usage patterns.…”

InteractOut: Leveraging Interaction Proxies as Input Manipulation Strategies for Reducing Smartphone Overuse

“…These preliminary comparisons show the potential of InteractOut to have a promising performance compared to prior works, while further study is needed to provide quantitative evidence, and it is difficult to directly compare their results because of the different study populations, durations, and other factors. Future work could also devise benchmarks and guidelines [76,77] to enable such comparisons across research studies, which is often difficult for human-subject experiments.…”