AI-Generated Survival Estimates: What are Physicians’ and Patients’ Criteria for Trust?
Saturday, October 14, 2023
10:30 AM – 11:45 AM ET
Location: Heron (Fourth Floor)
Rapid advancements in artificial intelligence and machine learning (AI/ML) in healthcare raise pressing questions about how much users should trust and rely on AI/ML systems, including for risk-based clinical decision making. The US and EU have proposed policy frameworks to advance trustworthy AI; however, few empirical insights guide our understanding of how trustworthiness criteria should be defined in real-world healthcare settings. As part of an AHRQ-funded study to develop a personalized risk calculator for patients with advanced heart failure after receiving a Left Ventricular Assist Device (LVAD), we present qualitative findings from interviews with patients and physicians regarding their criteria for trusting an AI-based algorithm estimating post-LVAD survival. Our findings reveal that patients and physicians share common rationale for hesitating to trust risk estimates due to the complexity and heterogeneity of risk factors. Patients and physicians also identified similar trust criteria, including understanding what factors influence the algorithm’s conclusions (explainability), how “right” the model is (accuracy), how it is tested and validated and on whom (validity; relevance), and whether its use is professionally endorsed (reputability). A minority said they would never trust the AI tool, citing the tool’s inability to account for immeasurable phenomena, including the role of “chance” (physicians) and hope, optimism and faith (patients) in influencing outcomes. Findings challenge the assumption that patients’ informational needs and criteria for trust in AI systems are less/fewer or qualitatively different from those of physicians, and highlight the centrality of training integrity and performance criteria as conditions for trust across stakeholders.
Benjamin Lang – Center for Medical Ethics and Health Policy – Baylor College of Medicine; Meghan Hurley – Center for Medical Ethics and Health Policy – Baylor College of Medicine; Jared Smith – Center for Medical Ethics and Health Policy – Baylor College of Medicine; Natalie Dorfman – Center for Medical Ethics and Health Policy – Baylor College of Medicine; J.S. blumenthal-barby – Center for Medical Ethics and Health Policy – Baylor College of Medicine