Identifying Women at Risk for Polycystic Ovary Syndrome Using a Mobile Health App: Virtual Tool Functionality Assessment

Abstract

Background Polycystic ovary syndrome (PCOS) is an endocrine disrupting disorder affecting about 10% of reproductive-aged women. PCOS diagnosis may be delayed several years and may require multiple physicians, resulting in lost time for risk-reducing interventions. Menstrual tracking apps are a potential tool to alert women of their risk while also prompting evaluation from a medical professional. Objective The primary objective of this study was to develop and pilot test the irregular cycle feature, a predictive model that generated a PCOS risk score, in the menstrual tracking app, Clue. The secondary objectives were to run the model using virtual test subjects, create a quantitative risk score, compare the feature’s risk score with that of a physician, and determine the sensitivity and specificity of the model before empirical testing on human subjects. Methods A literature review was conducted to generate a list of signs and symptoms of PCOS, termed variables. Variables were then assigned a probability and built into a Bayesian network. Questions were created based on these variables. A total of 9 virtual test subjects were identified using self-reported menstrual cycles and answers to the feature’s questions. Upon completion of the questionnaire, a Result Screen and Doctor’s Report summarizing the probability of having PCOS was displayed. This provided information about PCOS and data to facilitate diagnosis by a medical professional. To assess the accuracy of the feature, the same set of 9 virtual test subjects was assigned probabilities by the feature and the physician, who served as the gold standard. The feature recommended individuals with a score greater than or equal to 25% to follow-up with a physician. Differences between the feature and physician scores were evaluated using a t test and a Pearson correlation coefficient in 8 of the 9 virtual test subjects. A second iteration was conducted to assess the feature’s probability capabilities. Results The irregular cycle feature’s first iteration produced 1 false-positive compared with the physician score and had an absolute mean difference of 15.5% (SD 15.1%) among the virtual test subjects. The second iteration had 2 false positives compared with the physician score and had an absolute mean difference of 18.8% (SD 13.6%). The feature overpredicted the virtual test subjects’ risk of PCOS compared with the physician. However, a significant positive correlation existed between the feature and physician score (Pearson correlation coefficient=0.82; P=.01). The second iteration performed worse, with a Pearson correlation coefficient of 0.73 (P=.03). Conclusions The first iteration of the feature outperformed the second and better predicted the probability of PCOS. Although further research is needed with a more robust sample size, this pilot study indicates the potential value for developing a screening tool to prompt high-risk subjects to seek evaluation by a medical professional.

Assessing the accuracy of a digital symptom checker tool for suggestion of reproductive health conditions: a clinical vignettes study

“…Sensitivity and specificity in our symptom checkers was between 75-100%. The PCOS symptom checker evaluated by Rodriguez et al 56 reported 12-25% false-positive cases and no false-negative out of 8 cases tested. Our PCOS symptom checker had no false-negatives and three false-positive cases (12.5%) out of 24 cases tested.…”

“…If changes were suggested, the case would then be reviewed, edited, and approved by a third GP who would finalise the case. Twenty-four cases were created for each condition, in line with other single-condition or single-system symptom checker evaluations [55][56][57][58] .…”

“…Evaluations of single-condition symptom checkers include a study of 12 web-based symptom checkers for COVID-19 63 and a study of an app-based symptom checker for PCOS 56 . COVID-19 symptom checkers ranged widely in both sensitivity (14-94) and specificity (29-100), with only four symptom checkers having both sensitivity and specificity above 50% and two with both sensitivity and specificity above 75%.…”

“…Other vignette symptom checker evaluations have used between 3-400 cases for testing with single-condition or single-system evaluations (e.g. mental health, ophthalmology, PCOS) using fewer cases and multi-condition evaluations using larger numbers of cases [55][56][57]73,74 . Amongst the 400 vignettes published by Hammoud et al 74 , any single condition is only represented by at most five cases.…”

Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study

“…Evaluations of single-condition symptom checkers include a study of 12 web-based symptom checkers for COVID-19 [ 70 ] and a study of an app-based symptom checker for PCOS [ 62 ]. COVID-19 symptom checkers ranged widely in both sensitivity (14%-94%) and specificity (29%-100%), with only 4 symptom checkers having both sensitivity and specificity above 50% and 2 with both sensitivity and specificity above 75%.…”