BACKGROUND Previous studies have evaluated the accuracy of the diagnostics of electronic symptom checkers (ESCs) and triage using clinical case vignettes. Variation exists between different ESCs and the conditions being assessed, including the triage capabilities. There is a limited amount of evidence on live clinical patient user safety hazards associated with the use of ESCs, as safety has mainly been evaluated with the use of clinical vignettes. National Omaolo Digital services (Omaolo) in Finland consist of an ESC for various symptoms. Users can triage their symptoms with the help of ESC algorithms. As a result of completing the ESC, the user receives a recommendation for action and a service assessment with appropriate guidance regarding their health problems for a specific symptom. This allows users to be provided with healthcare services regardless of time and place. OBJECTIVE This study investigates how well triage by the ESC matches the triage of nurses among the chief symptom assessments available in Omaolo (anal region symptoms, cough, diarrhea, discharge from the eye or watery or reddish eye, headache, heartburn, knee symptom or injury, lower back pain or injury, oral health, painful or blocked ear, respiratory tract infection, sexually transmitted disease, shoulder pain or stiffness or injury, sore throat or throat symptom, urinary tract infection). In addition, this study assesses the safety of the Omaolo ESC. METHODS This is a clinical validation study in a real-life setting. It is a multicenter study in primary health centers across Finland. The included units were of the walk-in model of primary care, where no prior phone call or online contact is required. Upon arriving at the health center, users (patients) filled out the ESC questions and after that a nurse assessed their triage. Findings concerning 877 patients were analyzed. The triage recommendations by the ESC were matched with the triage assessments by nurses. A case was defined as safe if the recommendation for action given by the ESC was at most one degree of urgency less urgent than the triage assessment by the nurse concerning the same case. Critical cases were analyzed in detail. RESULTS The mean of the exact match for all symptom assessments was 53.7%. The mean of the exact match or overly conservative but suitable for all symptom assessments was 67.7%. Safe assessments by the ESC accounted for 97.6% of all assessments made. Critical cases were identified and further analyzed. In these 21 cases, no indication was found that patient safety had been compromised. CONCLUSIONS This study was the first to assess the accuracy and safety of the Omaolo ESC in the Finnish primary health care context. The findings indicate that the Omaolo ESC is safe compared to the assessment and triage of a triage nurse. INTERNATIONAL REGISTERED REPORT RR2-10.2196/41423

Background The national Omaolo digital social welfare and health care service of Finland provides a symptom checker, Omaolo, which is a medical device (based on Duodecim Clinical Decision Support EBMEDS software) with a CE marking (risk class IIa), manufactured by the government-owned DigiFinland Oy. Users of this service can perform their triage by using the questions in the symptom checker. By completing the symptom checker, the user receives a recommendation for action and a service assessment with appropriate guidance regarding their health problems on the basis of a selected specific symptom in the symptom checker. This allows users to be provided with appropriate health care services, regardless of time and place. Objective This study describes the protocol for the mixed methods validation process of the symptom checker available in Omaolo digital services. Methods This is a mixed methods study using quantitative and qualitative methods, which will be part of the clinical validation process that takes place in primary health care centers in Finland. Each organization provides a space where the study and the nurse triage can be done in order to include an unscreened target population of users. The primary health care units provide walk-in model services, where no prior phone call or contact is required. For the validation of the Omaolo symptom checker, case vignettes will be incorporated to supplement the triage accuracy of rare and acute cases that cannot be tested extensively in real-life settings. Vignettes are produced from a variety of clinical sources, and they test the symptom checker in different triage levels by using 1 standardized patient case example. Results This study plan underwent an ethics review by the regional permission, which was requested from each organization participating in the research, and an ethics committee statement was requested and granted from Pirkanmaa hospital district’s ethics committee, which is in accordance with the University of Tampere’s regulations. Of 964 clinical user–filled symptom checker assessments, 877 cases were fully completed with a triage result, and therefore, they met the requirements for clinical validation studies. The goal for sufficient data has been reached for most of the chief symptoms. Data collection was completed in September 2019, and the first feasibility and patient experience results were published by the end of 2020. Case vignettes have been identified and are to be completed before further testing the symptom checker. The analysis and reporting are estimated to be finalized in 2024. Conclusions The primary goals of this multimethod electronic symptom checker study are to assess safety and to provide crucial information regarding the accuracy and usability of the Omaolo electronic symptom checker. To our knowledge, this will be the first study to include real-life clinical cases along with case vignettes. International Registered Report Identifier (IRRID) DERR1-10.2196/41423

BACKGROUND National Omaolo digital services include symptom checkers in Finland. Users of these services can do their triage with the help of questions from the reasoning tree of the symptom checkers. As a result of completing the symptom checker, the user receives a recommendation of action and a service assessment with appropriate guidance regarding their health problems to a specific symptom. This allows users to be provided with healthcare services regardless of time and place. OBJECTIVE This study describes the protocol for the mixed methods validation process of the 15 symptom checkers available in Omaolo digital services. METHODS This is a three-part mixed methods study using quantitative and qualitative methods. The first (qualitative validation study) and second study (quantitative clinical validation study) will be part of the clinical validation process that takes place in primary healthcare centers in Finland. Each organization provides a space where the study and the nurse triage can be done, in order to include an unscreened target population of users. The primary healthcare units included in this study provide walk-in model services, where no mandatory prior phone call or contact is required. Finally, in the third part (validation with case vignettes study), the use of case vignettes will be incorporated to supplement the triage accuracy of some of the Omaolo symptom checkers. Vignettes are produced from a variety of clinical sources, including material used in the exams of medical students. These vignettes test symptom checkers, in different triage levels, using one standardized patient case example. RESULTS A total of 964 clinical user filled symptom checker assessments were available, of which, 877 cases meet the requirements for clinical validation studies. The goal of sufficient data has been reached for most of the symptom checkers. Data collection was completed at the end of December 2020, and the first feasibility and user experience results were published at the end of 2020. Case vignettes have been identified and are to be completed before further testing the symptom checkers. CONCLUSIONS The primary goals of this multimethod electronic symptom checker study are to assess safety, and to provide crucial information regarding the accuracy and usability of the Omaolo electronic symptom checkers. To the best of our knowledge this will be the first study to include real-life setting clinical cases along with case vignettes. Study results will be made publicly available through reports and open-access journal publications.