What Can Digital Disease Detection Learn from (an External Revision to) Google Flu Trends?

Santillana, Mauricio; Zhang, D. Wendong; Althouse, Benjamin M.; Ayers, John W.

doi:10.1016/j.amepre.2014.05.020

Cited by 163 publications

(184 citation statements)

References 40 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…This table shows that, despite the observed variation in the Google Trends data, ARGO is threefold more stable than the method of ref. 16, and still outperforms on average any other method.…”

Section: Resultsmentioning

confidence: 90%

“…These metrics include the root-mean-squared error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), correlation with estimation target, and correlation of increment with estimation target. For comparison, we calculated these accuracy metrics for (i) GFT estimates (accessed on July 11, 2015), (ii) estimates produced using the method of Santillana et al (6,16), (iii) estimates produced by combining GFT with a lag-3 autoregressive model, AR(3), as suggested in ref.…”

Section: Resultsmentioning

confidence: 99%

“…This theoretical framework contains, as a special case, the model developed in ref. 16. Our model not only achieves the goal of (i) dynamically incorporating new information from CDC reports as it becomes available and (ii) automatically selecting the most useful Google search queries for estimation as in ref.…”

Section: Our Contributionmentioning

confidence: 99%

“…Our model not only achieves the goal of (i) dynamically incorporating new information from CDC reports as it becomes available and (ii) automatically selecting the most useful Google search queries for estimation as in ref. 16, but also largely improves estimation by (iii) including the longterm cyclic information (seasonality) from past flu seasons on record as input variables and (iv) using a 2-y moving window (which immediately precedes the desired date of estimation) for the training period to capture the most recent changes in people's search patterns and time series behavior (33). Our methodology efficiently builds a prediction model from individual search frequency as well as the past records of ILI activity.…”

Section: Our Contributionmentioning

confidence: 99%

“…However, significant discrepancies between GFT's flu estimates and those measured by the Centers for Disease Control (CDC) in subsequent years led to considerable doubt about the value of digital disease detection systems (13). Although multiple articles have identified methodological flaws in GFT's original algorithm (14)(15)(16) and have led to incremental improvements (14,16) (see also googleresearch. blogspot.com/2014/10/google-flu-trends-gets-brand-new-engine.html), a statistical framework that is theoretically sound and capable of accurate estimation is still lacking.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Accurate estimation of influenza epidemics using Google search data via ARGO

Yang

Santillana

Kou

2015

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

361

443

View full text Add to dashboard Cite

show abstract

Section: Resultsmentioning

confidence: 90%

Section: Resultsmentioning

confidence: 99%

Section: Our Contributionmentioning

confidence: 99%

Section: Our Contributionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Accurate estimation of influenza epidemics using Google search data via ARGO

Yang

Santillana

Kou

2015

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

361

443

View full text Add to dashboard Cite

show abstract

Using “Big Data” to Optimize Public Health Outreach

Williams

Smith

2015

JAMA Dermatol

View full text Add to dashboard Cite

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

et al. 2023

Self Cite

View full text Add to dashboard Cite

ImportanceThe rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.ObjectiveTo evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions.Design, Setting, and ParticipantsIn this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit’s r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose “which response was better” and judged both “the quality of information provided” (very poor, poor, acceptable, good, or very good) and “the empathy or bedside manner provided” (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.ResultsOf the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P &lt; .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P &lt; .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P &lt; .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot.ConclusionsIn this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

show abstract

What Can Digital Disease Detection Learn from (an External Revision to) Google Flu Trends?

Cited by 163 publications

References 40 publications

Accurate estimation of influenza epidemics using Google search data via ARGO

Accurate estimation of influenza epidemics using Google search data via ARGO

Using “Big Data” to Optimize Public Health Outreach

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

Contact Info

Product

Resources

About