Data-Driven Diagnostics and the Potential of Mobile Artificial Intelligence for Digital Therapeutic Phenotyping in Computational Psychiatry

Washington, Peter; Park, Natalie; Srivastava, Parishkrita; Voss, Catalin; Kline, Aaron; Varma, Maya; Tariq, Qandeel; Kalantarian, Haik; Schwartz, Jessey; Patnaik, Ritik; Chrisman, Brianna; Stockham, Nate; Paskov, Kelley; Haber, Nick; Wall, Dennis P.

doi:10.1016/j.bpsc.2019.11.015

Cited by 76 publications

(68 citation statements)

References 109 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…More structured videos, such as those collected in home smartphone autism interventions [ 15 , 16 , 17 , 18 , 19 ], may yield more consistent video difficulty levels due to the standardization of collected videos. Mobile therapeutics in conjunction with crowdsourcing may be leveraged toward longitudinal outcome tracking of symptoms [ 7 ]. Testing more subsets of the crowd, partitioned not only by location but by a wide array of demographic factors, will reveal economical subsets of the crowd for remote behavioral video tagging.…”

Section: Discussionmentioning

confidence: 99%

“…Since autism consists of a largely behavioral phenotype, video data are a particularly powerful and rich means of capturing the range of social symptoms a child may exhibit in a fast and virtually cost-free manner. Accurate diagnoses and behavioral classifications have been inferred from categorical ordinal labels extracted by untrained humans from the short video clips [ 5 , 6 , 7 , 8 , 9 , 10 ], which are recorded by digital mobile and wearable interventions during use by the child or administering parent [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ]. Such a process can be scaled through crowdsourcing platforms, which allow distributed workers from around the globe to perform short on-demand tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

Washington

Leblanc

Dunlap

et al. 2020

JPM

Self Cite

View full text Add to dashboard Cite

Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers—defined as vetted members of popular crowdsourcing platforms—to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (p= 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

Washington

Leblanc

Dunlap

et al. 2020

JPM

Self Cite

View full text Add to dashboard Cite

show abstract

“…In anticipation of the widespread use of machine learning classifiers as detection tools for autism 45 , here we studied the impact of missing values on the performance of two previously published ASD classifiers, a logistic regression using 9 features (LR9) and an alternating decision tree model using 7 features (ADTree7), using a dataset of non-expert ratings of 140 YouTube child videos. We compared common univariate and multivariate feature imputation methods to general and dynamic feature replacement techniques.…”

Section: Discussionmentioning

confidence: 99%

Feature replacement methods enable reliable home video analysis for machine learning detection of autism

Leblanc

Washington

Varma

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

Autism Spectrum Disorder is a neuropsychiatric condition affecting 53 million children worldwide and for which early diagnosis is critical to the outcome of behavior therapies. Machine learning applied to features manually extracted from readily accessible videos (e.g., from smartphones) has the potential to scale this diagnostic process. However, nearly unavoidable variability in video quality can lead to missing features that degrade algorithm performance. To manage this uncertainty, we evaluated the impact of missing values and feature imputation methods on two previously published autism detection classifiers, trained on standard-of-care instrument scoresheets and tested on ratings of 140 children videos from YouTube. We compare the baseline method of listwise deletion to classic univariate and multivariate techniques. We also introduce a feature replacement method that, based on a score, selects a feature from an expanded dataset to fill-in the missing value. The replacement feature selected can be identical for all records (general) or automatically adjusted to the record considered (dynamic). Our results show that general and dynamic feature replacement methods achieve a higher performance than classic univariate and multivariate methods, supporting the hypothesis that algorithmic management can maintain the fidelity of video-based diagnostics in the face of missing values and variable video quality.

show abstract

“…Observational studies mainly comprise of cross-sectional, longitudinal and case-control designs [ 12 ], and have often been mistakenly considered as merely qualitative. Conversely, current observational analysis techniques allow one to collect quantitative data and to employ more sophisticated computational approaches for the systematic observation of behavior [ 13 ] This opportunity is highly relevant in the context of developmental and clinical research, where observational techniques have the great advantage of being almost completely non-invasive, or minimally invasive in many cases [ 3 , 14 , 15 ].…”

Section: Introductionmentioning

confidence: 99%

Quantifying the Child–Therapist Interaction in ASD Intervention: An Observational Coding System

et al. 2021

View full text Add to dashboard Cite

Background: Observational research plays an important part in developmental research due to its noninvasiveness. However, it has been hardly applied to investigate efficacy of the child–therapist interaction in the context of naturalistic developmental behavioral interventions (NDBI). In particular, the characteristics of child–therapist interplay are thought to have a significant impact in NDBIs in children with autism spectrum disorder (ASD). Quantitative approaches may help to identify the key features of interaction during therapy and could be translated as instruments to monitor early interventions. Methods: n = 24 children with autism spectrum disorder (ASD) were monitored from the time of the diagnosis (T0) and after about one year of early intervention (T1). A novel observational coding system was applied to video recorded sessions of intervention to extract quantitative behavioral descriptors. We explored the coding scheme reliability together with its convergent and predictive validity. Further, we applied computational techniques to investigate changes and associations between interaction profiles and developmental outcomes. Results: Significant changes in interaction variables emerged with time, suggesting that a favorable outcome is associated with interactions characterized by increased synchrony, better therapist’s strategies to successfully engage the child and scaffold longer, more complex and engaging interchanges. Interestingly, data models linked interaction profiles, outcome measures and response trajectories. Conclusion: Current research stresses the need for process measures to understand the hows and the whys of ASD early intervention. Combining observational techniques with computational approaches may help in explaining interindividual variability. Further, it could disclose successful features of interaction associated with better response trajectories or to different ASD behavioral phenotypes that could require specific dyadic modalities.

show abstract

Data-Driven Diagnostics and the Potential of Mobile Artificial Intelligence for Digital Therapeutic Phenotyping in Computational Psychiatry

Cited by 76 publications

References 109 publications

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

Feature replacement methods enable reliable home video analysis for machine learning detection of autism

Quantifying the Child–Therapist Interaction in ASD Intervention: An Observational Coding System

Contact Info

Product

Resources

About