Standards for Explainability: The Role of Stigma and Disease State in Patient Acceptance of Artificial Intelligence-Generated Care
Friday, October 13, 2023
3:15 PM – 4:30 PM ET
Location: Dover C (Third Floor)
An enduring focus in AI ethics on explainability of AI-generated health metrics raises questions about what patients should know about AI systems, including how AI systems generate their outputs. We present qualitative results from interviews conducted with patients and clinicians about patients’ informational needs regarding how AI-based risk estimates and classifications are derived. Our data comes from two NIH-funded studies: 1. An AHRQ-funded project to develop and integrate an AI-based personalized risk calculator into clinical decision support for patients considering LVAD for heart failure and 2. An NCATS-funded project examining ethical and practical challenges to translating perceptual computing (e.g. digital phenotyping; affective computing) into clinical care. Preliminary data shows that while patients across studies show a strong interest in knowing which factors contribute to algorithmic outputs, the degree to which patients are likely to accept or trust their integration into clinical decision-making depends in part on the disease type (e.g. physiological vs. psychiatric) addressed by AI outputs. Patients might be more inclined to question the validity of an output or feel that their health, well-being or identity is threatened when AI outputs address disease states that are potentially stigmatizing, such as psychosocial disorders. These findings contribute to our understanding of clinical and demographic factors influencing attitudes towards (and desire for) algorithmic explainability. They also highlight the potential need for a “higher bar” of explainability and interpretability of AI-derived risk estimates for certain conditions that patients may consider sensitive or potentially stigmatizing.
Kristin Kostick-Quenet – Assistant Professor, Center for Medical Ethics and Health Policy, Baylor College of Medicine; John Herrington – Psychologist, Child and Adolescent and Behavioral Sciences, Children's Hospital of Philadelphia; Eric Storch – Professor and Vice Chair of Psychology, Psychiatry and Behavioral Sciences, Baylor College of Medicine; Jennifer Blumenthal-Barby – Professor, Center for Medical Ethics and Health Policy, Baylor College of Medicine