Adapting Health AI Interfaces for User Expertise

New research from MIT and its collaborators is shedding light on a critical challenge in the deployment of artificial intelligence within the healthcare sector: the profound impact of user expertise on the effectiveness and interpretation of AI explainability tools.

The study, published in Nature Medicine, examined AI-powered diagnostic support for skin diseases. It found that while non-expert users experienced an improvement in diagnostic accuracy when aided by AI, this gain was largely attributable to a tendency to defer to the AI’s pronouncements. In contrast, primary care physicians demonstrated a distinct pattern, achieving their peak performance when presented with an AI prediction devoid of any explanation.

This divergence in outcomes underscores a complex interplay between AI, human cognition, and the design of user interfaces, particularly as AI tools become increasingly integrated into both clinical workflows and patient-facing applications.

Marzyeh Ghassemi, an associate professor in MIT’s Department of Electrical Engineering and Computer Science and a key figure in the research, emphasized the imperative for careful consideration in the design of health AI interfaces. “Good AI systems can indeed enhance performance in certain health settings, but this must be meticulously balanced against algorithmic deference, which can inadvertently lead to increased errors,” Ghassemi stated. “We recognize that both AI and explainability methodologies can foster automation bias in human users, and this anchoring effect is a crucial factor that must be addressed when architecting AI systems.”

The Interface’s Influence on Diagnosis

The core objective of Explainable AI (XAI) is to equip users with the rationale behind an AI model’s output, enabling them to critically assess its conclusions. Traditional approaches might involve highlighting salient regions within a medical image that contributed to a diagnosis, or presenting similar cases that support a given prediction. More recently, large language models (LLMs) have emerged as a novel avenue, capable of generating clear, natural-language explanations of an AI’s reasoning process, often tailored for a general audience.

The MIT-led investigation put several of these XAI techniques to the test. Participants were presented with medical images alongside an AI prediction for skin disease. The experimental setups varied: one interface provided only the prediction and a confidence level, omitting any explanation. Another displayed similar images, while a separate system employed heatmaps to pinpoint areas of interest. Crucially, researchers also evaluated LLM-generated textual explanations.

The study involved two distinct user groups. Non-experts were tasked with determining whether images of skin moles indicated malignancy. Clinicians, on the other hand, faced a more comprehensive challenge: providing a differential diagnosis for a range of dermatological conditions.

Non-Experts Showed Strongest Deference to Language-Based Explanations

Across all tested explainability methods, non-expert participants demonstrated an improvement in diagnostic accuracy. These AI tools were particularly effective in assisting users in correctly identifying non-cancerous moles.

The research also incorporated a fairness-constrained AI model, engineered to mitigate biases against individuals with darker skin tones. This specialized model not only enhanced accuracy but also reduced diagnostic disparities linked to skin pigmentation. However, this performance gain came with a significant caveat: non-experts exhibited an elevated reliance on the model’s recommendations. When the AI provided an incorrect output, it resulted in a more substantial detriment to their performance compared to the positive impact of correct outputs. “The enhanced performance observed in non-expert users stems from their greater dependence on the AI models. Consequently, when the model errs, the negative impact on performance is more pronounced than the positive effect when the model is accurate. We were able to train highly proficient AI models for this specific context,” explained Ghassemi.

LLM-generated explanations elicited the most pronounced deferential behavior. Participants exhibited a tendency to trust these explanations regardless of whether the underlying AI model’s output was accurate or flawed. Furthermore, the researchers noted that users found vague or generic explanations to be surprisingly convincing.

Users who received LLM assistance reported a greater sense of confidence, even when their answers were incorrect. This finding presents a substantial challenge for the design of consumer-facing diagnostic AI systems, where a seemingly authoritative and plausible textual explanation can lend undue credibility to an erroneous AI prediction.

Roxana Daneshjou, an assistant professor of biomedical data science and dermatology at Stanford University, highlighted the particular vulnerability of patients with limited medical knowledge. “These findings are critical as patients increasingly turn to AI for healthcare assistance,” Daneshjou commented. “Our research indicates that individuals with the least medical expertise are most susceptible to being misled when explainable AI models produce erroneous outputs.”

Primary Care Providers Interacted with AI Uniquely

In stark contrast, clinicians displayed a greater resilience to incorrect AI explanations. They were less likely to follow erroneous recommendations or explanations provided by the system. Their strongest performance was observed when they utilized a more streamlined interface, receiving only the AI model’s prediction without any accompanying explanatory text.

For clinicians, LLM-generated explanations resulted in the smallest accuracy improvements among all the explainability methods tested. This outcome does not suggest that explanations are without value in clinical practice. Instead, it highlights that an explanation format optimized for patients or novice users may not be suitable for trained professionals engaged in complex differential diagnosis.

Lead author Orson Xu, an assistant professor in Columbia University’s Department of Biomedical Informatics, elaborated on this crucial distinction: “The critical factor is how each user group interprets the explanation. A clinician already possesses a preliminary diagnosis and evaluates the AI’s output against their own professional training, thereby identifying potential flaws in a poorly constructed explanation. Conversely, a non-expert can leverage the same explanation to form their initial opinion, making them susceptible to an AI’s plausible, confident-sounding rationale that might lead them to an incorrect conclusion. The same AI tool can thus serve as a valuable asset for one user while becoming a potential liability for another.”

The study advocates against treating explainability as a one-size-fits-all interface component. It posits that a user’s baseline expertise fundamentally influences whether an explanation functions as a critical check on the AI model or, conversely, becomes a substitute for independent clinical judgment.

The Timing of AI Assistance Influences Automation Bias

The researchers also delved into the temporal aspect of AI assistance, investigating how the timing of exposure to AI explanations impacts user behavior. They observed that individuals exhibited greater deference to the AI when explanations were presented before they had the opportunity to formulate their own initial diagnosis. This finding offers a practical design insight: an interface could prompt users for their preliminary diagnostic hypothesis first, and then present the AI recommendation, potentially highlighting alternative conditions for consideration.

The study further revealed that users who demonstrated the highest degree of deference to AI were also the least proficient performers when undertaking the task without AI support. While these participants may stand to benefit the most from AI assistance, they also face the greatest risk when the AI model generates incorrect outputs.

The research compared human and AI performance across various disease presentations. AI systems consistently outperformed humans when symptoms were subtle. Conversely, humans exhibited significantly better performance when the medical image contained atypical symptoms or extraneous features.

For clinician-focused tools, direct model outputs that facilitate review against professional judgment are likely essential. For patient-facing applications, particular caution is warranted, especially regarding LLM explanations. This is particularly true when the AI system presents a confident, narrative-driven explanation for an incorrect recommendation, potentially leading vulnerable users down a dangerous path.