Sexual reproductive health chatbots: should we be so quick to throw artificial intelligence out with the bathwater?

Wasson, Elizabeth Jane; Driver, Katie; Hughes, Megan; Bailey, Jasmine

doi:10.1136/bmjsrh-2020-200823

Cited by 3 publications

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although these guidelines consider digital interventions in general, there is a growing interest in building a stronger evidence base on how SRH programs might leverage key technologies to improve their reach, engagement, and impact. One technology that has seen a substantial rise in use by SRH programs is chatbots [ 4 , 5 ].…”

Section: Introductionmentioning

confidence: 99%

Chatbots That Deliver Contraceptive Support: Systematic Review

Mills,

Mangone,

Lesh

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

Background A chatbot is a computer program that is designed to simulate conversation with humans. Chatbots may offer rapid, responsive, and private contraceptive information; counseling; and linkages to products and services, which could improve contraceptive knowledge, attitudes, and behaviors. Objective This review aimed to systematically collate and interpret evidence to determine whether and how chatbots improve contraceptive knowledge, attitudes, and behaviors. Contraceptive knowledge, attitudes, and behaviors include access to contraceptive information, understanding of contraceptive information, access to contraceptive services, contraceptive uptake, contraceptive continuation, and contraceptive communication or negotiation skills. A secondary aim of the review is to identify and summarize best practice recommendations for chatbot development to improve contraceptive outcomes, including the cost-effectiveness of chatbots where evidence is available. Methods We systematically searched peer-reviewed and gray literature (2010-2022) for papers that evaluated chatbots offering contraceptive information and services. Sources were included if they featured a chatbot and addressed an element of contraception, for example, uptake of hormonal contraceptives. Literature was assessed for methodological quality using appropriate quality assessment tools. Data were extracted from the included sources using a data extraction framework. A narrative synthesis approach was used to collate qualitative evidence as quantitative evidence was too sparse for a quantitative synthesis to be carried out. Results We identified 15 sources, including 8 original research papers and 7 gray literature papers. These sources included 16 unique chatbots. This review found the following evidence on the impact and efficacy of chatbots: a large, robust randomized controlled trial suggests that chatbots have no effect on intention to use contraception; a small, uncontrolled cohort study suggests increased uptake of contraception among adolescent girls; and a development report, using poor-quality methods, suggests no impact on improved access to services. There is also poor-quality evidence to suggest increased contraceptive knowledge from interacting with chatbot content. User engagement was mixed, with some chatbots reaching wide audiences and others reaching very small audiences. User feedback suggests that chatbots may be experienced as acceptable, convenient, anonymous, and private, but also as incompetent, inconvenient, and unsympathetic. The best practice guidance on the development of chatbots to improve contraceptive knowledge, attitudes, and behaviors is consistent with that in the literature on chatbots in other health care fields. Conclusions We found limited and conflicting evidence on chatbots to improve contraceptive knowledge, attitudes, and behaviors. Further research that examines the impact of chatbot interventions in comparison with alternative technologies, acknowledges the varied and changing nature of chatbot interventions, and seeks to identify key features associated with improved contraceptive outcomes is needed. The limitations of this review include the limited evidence available on this topic, the lack of formal evaluation of chatbots in this field, and the lack of standardized definition of what a chatbot is.

show abstract

Section: Introductionmentioning

confidence: 99%

Chatbots That Deliver Contraceptive Support: Systematic Review

Mills,

Mangone,

Lesh

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

show abstract

Sexual and reproductive health rights in the era of artificial intelligence

Bolarinwa,

Adebisi,

Ajayi

et al. 2024

The Lancet

View full text Add to dashboard Cite

Real-World Evaluation of Artificial Intelligence (AI) Chatbots for Providing Sexual Health Information: A Consensus Study Using Clinical Queries

Latt,

Aung,

Htaik

et al. 2024

Preprint

View full text Add to dashboard Cite

Introduction Artificial Intelligence (AI) chatbots could potentially provide information on sensitive topics, including sexual health, to the public. However, their performance compared to human clinicians and across different AI chatbots, particularly in the field of sexual health, remains understudied. This study evaluated the performance of three AI chatbots - two prompt-tuned (Alice and Azure) and one standard chatbot (ChatGPT by OpenAI) - in providing sexual health information, compared to human clinicians. Methods We analysed 195 anonymised sexual health questions received by the Melbourne Sexual Health Centre phone line. A panel of experts in a blinded order using a consensus-based approach evaluated responses to these questions from nurses and the three AI chatbots. Performance was assessed based on overall correctness and five specific measures: guidance, accuracy, safety, ease of access, and provision of necessary information. We conducted subgroup analyses for clinic-specific (e.g., opening hours) and general sexual health questions and a sensitivity analysis excluding questions that Azure could not answer. Results Alice demonstrated the highest overall correctness (85.2%; 95% confidence interval (CI), 82.1%-88.0%), followed by Azure (69.3%; 95% CI, 65.3%-73.0%) and ChatGPT (64.8%; 95% CI, 60.7%-68.7%). Prompt-tuned chatbots outperformed the base ChatGPT across all measures. Azure achieved the highest safety score (97.9%; 95% CI, 96.4%-98.9%), indicating the lowest risk of providing potentially harmful advice. In subgroup analysis, all chatbots performed better on general sexual health questions compared to clinic-specific queries. Sensitivity analysis showed a narrower performance gap between Alice and Azure when excluding questions Azure could not answer. Conclusions Prompt-tuned AI chatbots demonstrated superior performance in providing sexual health information compared to base ChatGPT, with high safety scores particularly noteworthy. However, all AI chatbots showed susceptibility to generating incorrect information. These findings suggest the potential for AI chatbots as adjuncts to human healthcare providers for providing sexual health information while highlighting the need for continued refinement and human oversight. Future research should focus on larger-scale evaluations and real-world implementations.

show abstract

Sexual reproductive health chatbots: should we be so quick to throw artificial intelligence out with the bathwater?

Abstract: Contributors EJW wrote the bulk of the letter, while the other authors aided with the formulation of ideas and proofreading. Patient consent for publication Not required.

Cited by 3 publications

References 4 publications

Chatbots That Deliver Contraceptive Support: Systematic Review

Chatbots That Deliver Contraceptive Support: Systematic Review

Sexual and reproductive health rights in the era of artificial intelligence

Real-World Evaluation of Artificial Intelligence (AI) Chatbots for Providing Sexual Health Information: A Consensus Study Using Clinical Queries

Contact Info

Product

Resources

About