No abstract

Would you allow an AI agent to make decisions on your behalf? If the answer is “not always,” the next question becomes “in what circumstances”? Answering this question requires human users to be able to assess an AI agent—and not just with overall pass/fail assessments or statistics. Here users need to be able to localize an agent’s bugs so that they can determine when they are willing to rely on the agent and when they are not. After-Action Review for AI (AAR/AI), a new AI assessment process for integration with Explainable AI systems, aims to support human users in this endeavor, and in this article we empirically investigate AAR/AI’s effectiveness with domain-knowledgeable users. Our results show that AAR/AI participants not only located significantly more bugs than non-AAR/AI participants did (i.e., showed greater recall) but also located them more precisely (i.e., with greater precision). In fact, AAR/AI participants outperformed non-AAR/AI participants on every bug and were, on average, almost six times as likely as non-AAR/AI participants to find any particular bug. Finally, evidence suggests that incorporating labeling into the AAR/AI process may encourage domain-knowledgeable users to abstract above individual instances of bugs; we hypothesize that doing so may have contributed further to AAR/AI participants’ effectiveness.

In what circumstances would you want this AI to make decisions on your behalf?" We have been investigating how to enable a user of an Artificial Intelligence-powered system to answer questions like this through a series of empirical studies, a group of which we summarize here. We began the series by (a) comparing four explanation configurations of saliency explanations and/or reward explanations. From this study we learned that, although some configurations had significant strengths, no one configuration was a clear "winner." This result led us to hypothesize that one reason for the low success rates Explainable AI (XAI) research has in enabling users to create a coherent mental model is that the AI itself does not have a coherent model. This hypothesis led us to (b) build a model-based agent, to compare explaining it with explaining a model-free agent. Our results were encouraging, but we then realized that participants' cognitive energy was being sapped by having to create not only a mental model, but also a process by which to create that mental model. This realization led us to (c) create such a process (which we term After-Action Review for AI or "AAR/AI") for them, integrate it into the explanation environment, and compare participants' success with AAR/AI scaffolding vs without it. Our AAR/AI studies' results showed that AAR/AI participants were more effective assessing the AI than non-AAR/AI participants, with significantly better precision and significantly better recall at finding the AI's reasoning flaws.

No abstract

were identified (first diagnosis = index date). Descriptive analysis was conducted to evaluate demographics, comorbidities and concomitant medications 6-months prior to or at index and diagnoses (symptoms/complications) and healthcare resource utilization 14-days prior to or at index. Results: The final sample comprised 553,728 patients with a COVID-19 diagnosis. Over half (54.5%) were female. Mean (SD) age at index was 57.4 (18.8) years; 18.0% were 60-69 and 30.3% were 70+. Half (49.5%) were located in the Northeast, including 25.0% in New York. Commonly observed comorbidities included hypertension (34.4%), type 2 diabetes (19.9%), dyslipidemia (18.9%), kidney failure/disease (13.6%), and asthma/COPD (12.1%). The top symptoms were cough (18.2%), shortness of breath/difficulty breathing (17.3%), and fever (15.5%); however, 64.2% had none of these three symptoms. Serious complications included pneumonia (23.4%), acute hypoxemic respiratory failure (10.8%), and sepsis/ septic shock (7.0%). One-third (34.5%) had an ER visit while one-fifth (22.6%) had a COVID-19 test within 14 days of diagnosis. Conclusions: COVID-19 is associated with substantial comorbidity and clinical burden in the US. Longer-term data, including clinical outcomes, is needed to better understand characteristics of patients affected with COVID-19.