Objectives: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. Method: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. Results: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. Conclusion: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

BackgroundThe EUprimecare project-team assessed the perception of primary health care (PHC) professionals and patients on quality of organization of PHC systems in the participating countries: Estonia, Finland, Germany, Hungary, Italy, Lithuania and Spain. This article presents the aggregated opinions, expectations and priorities of patients and professionals along some main dimensions of quality in primary health care, such as access, equity, appropriateness and patient- centeredness.MethodsThe focus group technique was applied in the study as a qualitative research method for exploration of attitudes regarding the health care system and health service. Discussions were addressing the topics of: general aspects of quality in primary health care; possibilities to receive/provide PHC services based on both parties needs; determinant factors of accessibility to PHC services; patient centeredness. The data sets collected during the focus group discussions were evaluated using the method of thematic analysis.ResultsThere were 14 focus groups in total: a professional and a patient group in each of the seven partner countries. Findings of the thematic analysis were summarized along the following dimensions: access and equity, appropriateness (coordination, continuity, competency and comprehensiveness) and patient centeredness.ConclusionsThis study shows perceptions and views of patients in interaction with PHC and opinion of professionals working in PHC. It serves as source of criteria with relevance to everyday practice and experience. The criteria mentioned by patients and by health care professionals which were considered determining factors of the quality in primary care were quite similar among the investigated countries. However, the perception and the level of tolerance regarding some of the criteria differed among EUprimecare countries. Among these dissimilar criteria we especially note the gate-keeping role of GPs, the importance of nurses' competency and the acceptance of waiting times. The impact of waiting time on patient satisfaction is obvious; the influence of equity and access to PHC services are more dependent on the equal distribution of settings and doctors in urban and rural area. Foreseen shortage of doctors is expected to have a substantial influence on patient satisfaction in the near future.

Background Electronic health records (EHRs) are an elementary part of the work of registered nurses (RNs) in healthcare. RNs are the largest group of healthcare workers, and their experiences with EHRs and their informatics competence play a crucial role in a fluent workflow. The present study examined EHR usability factors and nurses’ informatics competence factors related to self-reported time pressure and psychological distress. Methods A nationwide survey was conducted for working-age registered nurses in 2017. The study sample included 3607 nurses (5% men) in Finland. The association of age, sex, employment sector, EHR usability factors, and nurses’ informatics competence factors with time pressure and psychological distress were examined with analyses of covariance. Results The EHR usability factors that were associated with high time pressure were low EHR reliability and poor user-friendliness. Regarding the nurses’ informatics competence factors, only low e-Care competence was associated with time pressure. Of the EHR usability factors, low EHR reliability and low support for cooperation were associated with high psychological distress. Of the nurses’ informatics competence factors, low e-Care competence was associated with high psychological distress. Conclusions Unreliability and poor user-friendliness of EHRs seem to be prominent sources of time pressure and psychological distress among registered nurses. User-friendly EHR systems and digital tools in healthcare are needed. Nurses’ competence to use eHealth tools to tailor patient care should be strengthened through organizational and regional actions. For example, house rules about how to use eHealth tools and instructions on common practices in cooperation with other organizations could be useful.

BackgroundPoorly functioning, time-consuming, and inadequate information systems are among the most important work-related psychosocial factors causing stress in physicians. The present study examined the trend in the perceived stress that was related to information systems (SRIS) among Finnish physicians during a nine-year follow-up. In addition, we examined the associations of gender, age, employment sector, specialization status, leadership position, on-call burden, and time pressure with SRIS change and levels.MethodsA longitudinal design with three survey data collection waves (2006, 2010 and 2015) based on a random sample of Finnish physicians in 2006 was used. The study sample included 1095 physicians (62.3% women, mean age 54.4 years) who provided data on SRIS in every wave. GLM repeated measures analyses were used to examine the associations between independent variables and the SRIS trend during the years 2006, 2010, and 2015.ResultsSRIS increased during the study period. The estimated marginal mean of SRIS in 2006 was 2.80 (95% CI = 2.68–2.92) and the mean increase was 0.46 (95% CI = 0.30–0.61) points from 2006 to 2010 and 0.25 (95% CI = 0.11–0.39) points from 2010 to 2015. Moreover, our results show that the increase was most pronounced in primary care, whereas in hospitals SRIS did not increase between 2010 and 2015. SRIS increased more among those in a leadership position. On-call duties and high time-pressures were associated with higher SRIS levels during all waves.ConclusionsChanging, difficult, and poorly functioning information systems (IS) are a prominent source of stress among Finnish physicians and this perceived stress continues to increase. Organizations should implement arrangements to ease stress stemming from IS especially for those with a high workload and on-call or leadership duties. To decrease IS-related stress, it would be important to study in more detail the main IS factors that contribute to SRIS. Earlier studies indicate that the usability and stability of information systems as well as end-user involvement in system development and work-procedure planning may be significant factors.

Sairaanhoitajien kokemuksia potilas- ja asiakastietojärjestelmistä kartoitettiin nyt toiseen kertaan valtakunnallisella kyselytutkimuksella. Tutkimuksen tavoitteena on tuottaa tietoa kansallisessa Tieto hyvinvoinnin ja uudistuvien palvelujen tukena – Sote-tieto hyötykäyttöön 2020 -strategiassa asetettujen tavoitteiden saavuttamisesta yhden osa-alueen (Kyvykkäille käyttäjille fiksut järjestelmät) osalta sairaanhoitajien näkökulmasta. Näitä strategian osatavoitteita mukailevat tutkimuskysymykset ovat: Miten asiakas- ja potilastietojärjestelmät tukevat hoitajien työtehtäviä, yhteistyötä ja tiedonkulkua? Minkälaiseksi hoitajat arvioivat nykyisen käyttökokemuksensa, koulutuksensa, perehdyttämisen toimintatapojen muutokseen sekä lisäkoulutustarpeensa? Miten hyödylliseksi hoitajat arvioivat asiakas- ja potilastietojärjestelmät työssään? Tutkimusaineisto kerättiin sähköisellä kyselyllä hyödyntäen Sairaanhoitajaliiton, Tehyn sekä Akavan sairaanhoitajat ja Taja ry:n (TAJA) jäsenrekistereitä keväällä 2020. Kyselyyn vastasi hyväksyttävästi 3 610 sairaanhoitajaa, kätilöä tai terveydenhoitajaa, jotka toimivat julkisessa terveydenhuollossa, sosiaalihuollossa tai yksityisellä sektorilla. Vastaajat olivat kokeneita asiakas- ja potilastietojärjestelmien käyttäjiä. Valtaosa heistä kirjautui vähintään yhteen järjestelmään päivittäin, osa jopa viiteen tai sitä useampaan järjestelmään. Tulosten mukaan vastaajat eivät olleet tyytyväisiä järjestelmien tukeen niiden käytön vaatiman pitkän perehdytyksen vuoksi. Tiedonkulkuun hoitajien välillä omassa organisaatiossa oltiin tyytyväisiä, mutta tiedonkulussa eri organisaatioissa toimivien hoitajien välillä sekä hoitajien ja potilaiden välillä koettiin puutteita. Enemmistö vastaajista koki tietotekniset perustaitonsa, taitonsa tehdä kirjaukset asiakas- tai potilastietojärjestelmään, taitonsa tehdä potilaan hoidon kirjaus hoitotyön prosessin mukaisesti sekä tietosuojan ja tietoturvan periaatteiden hyödyntämisensä päivittäisessä työssä hyviksi tai erinomaisiksi kaikissa toimintaympäristöissä. Yli puolet vastaajista ei ilmoittanut lisäkoulutustarpeita. Kuitenkin hoitajat toivoivat työnantajiltaan enemmän täydennyskoulutusta. Vastaajat kokivat ongelmalliseksi järjestelmien kyvyn tai toiminnallisuuden koostaa yhteenvetonäkymiä. Yhteenvetona voidaan todeta sairaanhoitajilla olevan hyvät tai erinomaiset taidot käyttää asiakas- ja potilastietojärjestelmiä, mutta perehdytystä tarvitaan tukemaan digitaalisten palvelujen tuottamissa työprosesseissa. Hoidon jatkuvuus, laatu ja potilasturvallisuus ovat erityisesti hyötyjä, joita tietojärjestelmien käytöllä jo saavutetaan.