Is the Algorithm Good in a Bad World, or Has It Learned to be Bad? The Ethical Challenges of “Locked” Versus “Continuously Learning” and “Autonomous” Versus “Assistive” AI Tools in Healthcare

What happens when a patient-interfacing conversational artificial intelligence system (CAI)—AI that combines natural language understanding, processing, and machine-learning models to autonomously emulate human cognition and engagement—says something triggering, is biased or outright discriminatory in its conversation, or makes the wrong association and leads a vulnerable patient away from appropriate conversational therapy and into acceleration of their mental illness? Should healthcare systems allow their AI tools to autonomously evolve, or should they “lock” their algorithms, so safety in general and mitigation of the above issues specifically can be validated? If the former, who is responsible for ongoing validation and monitoring of the AI system to ensure its algorithmic output remains appropriate? If the latter, when and how often should updates happen, given that locked algorithms performance degrades overtime? How should clinicians and healthcaresystems evaluate and mitigate the risks posed by those choices? Are there clinical contexts in which AI should not be allowed to function autonomously and only be assistive? While, increasingly, machine learning tools are being designed for healthcare contexts (MLHC), these applications are still novel and struggle to “cross the chasm,” or demonstrate clinical impact. Given the limited in situ MLHC case studies, potential ethical concerns with envisioned MLHC still need to be largely extrapolated from ethical problems that have arisen in non-healthcare contexts (Marwaha and Kevlar 2022). Previously, we put forward a framework for identifying ethical challenges with MLHC through considering ethical issues raised by each design decision (and the questions designers needed to answer) along the entire conceptual pipeline from initial envisioning of an MLHC through deployment and clinical use (Char et al. 2020). We subsequently built on that framework to consider how to best operationalize studying ethical issues with regulatory implications (Abr amoff, Tobey, and Char 2020). In this commentary, we emphasize two important design considerations for CAI with significant ethical implications on patient safety, efficiency, and responsibility—adding to Sedlakova and Trachsel's (2023) discussion; whether MLHC tools (of which CAI is a subset) should operate as a locked or a continuously learning system, and whether they should be deployed as autonomous or assistive tools. As MLHC is early in adoption and clinical testing, the most pressing ethical concerns cannot be separated from the ‘cross the chasm’ problem: do MLHC actually work and improve clinical care?