The challenge of informed consent in AI-supported healthcare

This presentation addresses the demands that AI-assisted clinical decision making places on the doctrine of informed consent. The latter requires that patients are fully informed about the nature of a medical procedure or intervention, the potential risks and benefits, and any available alternatives. It aims to respect patient autonomy, promote trust in the patient-provider relationship, and safeguard against unethical practices. There are some well-known challenges to achieving genuinely informed consent, including a lack of understanding due, for instance, to the use of complex jargon and varying levels of literacy, and the existence of power asymmetries between patients and providers. Yet the integration of AI into clinical decision-making introduces an additional layer of complexity to these challenges. In many cases, the reasoning behind an AI-generated recommendation is difficult to explain because of the complex computational architectures that underpin these systems. This “black box” problem creates a challenge for informed consent: if clinicians themselves cannot fully understand how a system arrives at a particular recommendation, it becomes difficult to communicate the basis of that recommendation to patients in a meaningful way. The problem is further complicated by the fact that explainability in AI is not always straightforward to achieve. In some contexts, making a model more interpretable may come at the cost of reduced accuracy, potentially leading to worse medical outcomes for patients. Even when developers provide information about their algorithms or offer post hoc explanations, these disclosures may omit key data features or weightings that influence the final output, leaving the underlying reasoning difficult to interpret. As a result, clinicians may face legal and ethical dilemmas when advising patients about the use of AI-assisted recommendations in their care, particularly where the reliability of a system must be justified without a clear understanding of how its outputs are produced. Given this background, the presentation will examine the literature on explainable AI in order to assess what kinds of explanations current systems can realistically provide and whether these explanations satisfy the requirements of informed consent. The analysis will also consider possible interpretations of the doctrine of informed consent to accommodate the growing role of AI in clinical practice: for instance, by reconsidering the type of information required for consent to be informed, such as providing higher-level information about the reliability or success rates of an AI system rather than detailed explanations of its internal functioning.