FolioStart free

Computer Science · Literature

Research papers on Explainable AI in healthcare

Recent and highly-cited academic work on explainable ai in healthcare, gathered from Semantic Scholar, CrossRef and OpenAlex.

Search all 200M+ papers on this topic, free →Or track new explainable ai in healthcare papers automatically as they publish

  1. Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges

    Qaiser Abbas, Woo-Min Jeong, Seung Won Lee · 2025 · Healthcare · 268 citations

    Background: Theintegration of artificial intelligence (AI) into clinical decision support systems (CDSSs) has significantly enhanced diagnostic precision, risk stratification, and treatment planning. AI models remain a barrier to clinical adoption, emphasizing the critical role of explainable AI (XAI). Methods: This systematic meta-analysis synthesizes findings from 62 peer-reviewed studies published between 2018 and 2025, examining the use of XAI methods within CDSSs across various clinical domains, including radiology, oncology, neurology, and critical care. Model-agnostic techniques such as visualization models like Gradient-weighted Class Activation Mapping (Grad-CAM) and attention mecha

    Save this paper

  2. Co-design of Human-centered, Explainable AI for Clinical Decision Support

    Cecilia Panigutti, Andrea Beretta, Daniele Fadda, et al. · 2023 · ACM Transactions on Interactive Intelligent Systems · 120 citations

    eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (D

    Save this paper

  4. Explainability in medicine in an era of AI-based clinical decision support systems

    Robin L Pierce, Wim Van Biesen, Daan Van Cauwenberge, et al. · 2022 · Frontiers in Genetics · 94 citations

    The combination of "Big Data" and Artificial Intelligence (AI) is frequently promoted as having the potential to deliver valuable health benefits when applied to medical decision-making. However, the responsible adoption of AI-based clinical decision support systems faces several challenges at both the individual and societal level. One of the features that has given rise to particular concern is the issue of explainability, since, if the way an algorithm arrived at a particular output is not known (or knowable) to a physician, this may lead to multiple challenges, including an inability to evaluate the merits of the output. This "opacity" problem has led to questions about whether physician

    Save this paper

  5. Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis

    Mozhgan Salimparsa, K. Sedig, Dan Lizotte, et al. · 2025 · Informatics · 47 citations

    While Artificial Intelligence (AI) promises significant enhancements for Clinical Decision Support Systems (CDSSs), the opacity of many AI models remains a major barrier to clinical adoption, primarily due to interpretability and trust challenges. Explainable AI (XAI) seeks to bridge this gap by making model reasoning understandable to clinicians, but technical XAI solutions have too often failed to address real-world clinician needs, workflow integration, and usability concerns. This study synthesizes persistent challenges in applying XAI to CDSS—including mismatched explanation methods, suboptimal interface designs, and insufficient evaluation practices—and proposes a structured, user-cent

    Save this paper

  6. A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

    A. Gambetti, Qi-Wei Han, Hong Shen, et al. · 2025 · ArXiv · 22 citations

    Explainable Artificial Intelligence (XAI) is essential for the transparency and clinical adoption of Clinical Decision Support Systems (CDSS). However, the real-world effectiveness of existing XAI methods remains limited and is inconsistently evaluated. This study conducts a systematic PRISMA-guided survey of 31 human-centered evaluations (HCE) of XAI applied to CDSS, classifying them by XAI methodology, evaluation design, and adoption barrier. Our findings reveal that most existing studies employ post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, typically assessed through small-scale clinician studies. The results show that over 80% of the studies adopt post-hoc, model-agnostic

    Save this paper

  7. Explainable AI in Healthcare: Systematic Review of Clinical Decision Support Systems

    N. Aziz, A. Manzoor, M. D. Mazhar Qureshi, et al. · 2024 · 18 citations

    This systematic review examines the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in Clinical Decision Support Systems (CDSS), highlighting significant advancements and identifying persistent challenges. Utilising the PRISMA protocol, we searched major indexed databases such as Scopus, Web of Science, PubMed, and the Cochrane Library, to analyse publications from January 2000 to April 2024. This timeframe captures the progressive integration of XAI in CDSS, offering a historical and technological overview. The review covers the datasets, application areas, machine learning models, explainable AI methods, and evaluation strategies for multiple XAI methods. Analy

    Save this paper

  8. Multi-task reinforcement learning and explainable AI-Driven platform for personalized planning and clinical decision support in orthodontic-orthognathic treatment

    Zhi-Yuan Li, Li-Wei Wang · 2025 · Scientific Reports · 14 citations

    This study presents a novel clinical decision support platform for orthodontic-orthognathic treatment that integrates multi-task reinforcement learning with explainable artificial intelligence. The platform addresses the challenges of personalized treatment planning in complex dentofacial deformities by formulating treatment as a sequential decision-making process optimizing multiple clinical objectives simultaneously. We developed a comprehensive framework comprising: (1) a multi-task reinforcement learning core with specialized state-action representations for craniofacial structures; (2) complementary explainable AI components that render complex model decisions interpretable within clini

    Save this paper

  9. Explainable AI for Trustworthy Clinical Decision Support: A Case-Based Reasoning System for Nursing Assistants

    Yen-Ku Liu, Yun-Cheng Tsai · 2024 · 2024 IEEE International Conference on Big Data (BigData) · 8 citations

    The increasing complexity of patient data and the need for reliable, transparent decision-making in healthcare drive the demand for advanced AI systems. This paper introduces a Clinical Decision Support System (CDSS) integrating Explainable AI (XAI) with Retrieval-Augmented Generation (RAG), designed to offer transparent, real-time, and evidence-based recommendations. HIPAA-compliant and grounded in historical case studies, the system facilitates trustworthy decision-making in high-pressure environments for nursing assistants. Although preliminary, simulations indicate the system’s potential to reduce cognitive load and enhance decision accuracy, with future clinical trials planned for compr

    Save this paper

  10. Current state and promise of user-centered design to harness explainable AI in clinical decision-support systems for patients with CNS tumors

    Eric W. Prince, D. M. Mirsky, Todd C. Hankinson, et al. · 2025 · Frontiers in Radiology · 7 citations

    In neuro-oncology, MR imaging is crucial for obtaining detailed brain images to identify neoplasms, plan treatment, guide surgical intervention, and monitor the tumor's response. Recent AI advances in neuroimaging have promising applications in neuro-oncology, including guiding clinical decisions and improving patient management. However, the lack of clarity on how AI arrives at predictions has hindered its clinical translation. Explainable AI (XAI) methods aim to improve trustworthiness and informativeness, but their success depends on considering end-users’ (clinicians') specific context and preferences. User-Centered Design (UCD) prioritizes user needs in an iterative design process, invo

    Save this paper

  11. Explainable AI for Diabetes Risk Assessment: Enhancing Clinical Decision Support with Interpretable Machine Learning Models

    C. S. Reddy, Mohan Annamalai · 2025 · 2025 5th International Conference on Soft Computing for Security Applications (ICSCSA) · 7 citations

    This paper presents a comprehensive study on the application of Explainable Artificial Intelligence (XAI) for diabetes risk assessment, focusing on the interpretability of machine learning models in clinical decision support systems. While machine learning has demonstrated high accuracy in predicting diabetes, the lack of transparency in decision-making processes limits its adoption in healthcare. We employ interpretable models and model-agnostic explanation techniques such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) to enhance the understanding of predictive outcomes. Using real-world datasets including the PIMA Indians Diabetes Dataset

    Save this paper

  12. Enhancing AI Clinical Decision Support Trust: Design Workshop Insights from General Practitioners

    R. A. D. L. M. K. Ranwala, André Queiroz de Andrade · 2025 · Studies in health technology and informatics · 7 citations

    Artificial Intelligence (AI) predictive models are increasingly integrated into Clinical Decision Support Systems (CDSS). However, real-world implementation is lagging due to a lack of trust and acceptance by healthcare providers. To investigate factors related to trust and acceptance of AI-based CDSS, we conducted a workshop with General Practitioners to explore user, model, and organizational factors that could affect trust and recommendation acceptance. The workshop discussions revealed that while explainability is crucial for understanding AI decisions, current explainable AI (XAI) visualizations appeared to provide limited value to clinicians without technical backgrounds. Participants

    Save this paper

  13. Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives

    B. V. Van Dort, Thomas Engelsma, Pamela Sneekes, et al. · 2026 · PLOS Digital Health · 7 citations

    Explainable Artificial Intelligence (XAI) has the potential to enhance clinical decision support (CDS) systems however, it remains unclear how XAI systems are perceived by healthcare professionals in hospital settings, and if new challenges arise as a result of explanations. This scoping review aimed to understand healthcare professionals’ perceptions of CDS systems with XAI in the hospital setting; specifically, the drivers of acceptance and use, explainability needs, and design preferences. Databases were searched. Studies were included if they reported qualitative findings on health professionals’ perceptions of XAI-enabled CDS systems used in hospital settings. MEDLINE, Embase, and Web o

    Save this paper

  14. Leveraging ChatGPT and explainable AI for enhancing clinical decision support

    Radwa El Shawi, Leila Jamel · 2025 · Scientific Reports · 6 citations

    Large language models (LLMs) excel in many natural language processing tasks. However, their direct application to tabular, domain-specific clinical data remains challenging, as they lack innate mechanisms for reasoning over structured numerical features. This paper presents HealthAI-Prompt, a novel framework that systematically adapts LLMs for tabular clinical decision-making-specifically, predicting diabetes risk-through contextual prompts that combine detailed task descriptions with domain knowledge. Our domain knowledge integration leverages insights from high-performing machine learning models optimized via automated machine learning (AutoML) technique, together with local explanations

    Save this paper

  15. AXAI-CDSS: An Affective Explainable AI-Driven Clinical Decision Support System for Cannabis Use

    Tongze Zhang, Tammy Chung, Anind K. Dey, et al. · 2025 · 2025 International Conference on Activity and Behavior Computing (ABC) · 4 citations

    As cannabis use has increased in recent years, researchers have come to rely on sophisticated machine learning models to predict cannabis use behavior and its impact on health. However, many artificial intelligence (AI) models lack transparency and interpretability due to their opaque nature, limiting their trust and adoption in real-world medical applications, such as clinical decision support systems (CDSS). To address this issue, this paper enhances algorithm explainability underlying CDSS by integrating multiple Explainable Artificial Intelligence (XAI) methods and applying causal inference techniques to clarify the models’ predictive decisions under various scenarios. By providing deepe

    Save this paper

  17. User-Centered Methods in Explainable AI Development for Hospital Clinical Decision Support: A Scoping Review.

    B. V. Van Dort, Thomas Engelsma, S. Medlock, et al. · 2025 · Studies in health technology and informatics · 3 citations

    Explainable Artificial Intelligence (XAI) offers promising advancements in enhancing transparency and usability of AI-based Clinical Decision Support Systems (CDSS) in healthcare settings. These tools aim to improve clinical outcomes by assisting with diagnosis, treatment planning, and risk prediction. However, integrating XAI into clinical workflows requires effective involvement of healthcare professionals to ensure that the explanations provided by these tools are comprehensible, relevant, and actionable. This scoping review aimed to investigate how (potential) end users were involved in the design and development of XAI-based CDSS for hospitals. A systematic search of Medline, Embase, an

    Save this paper

  20. Enhancing Clinical Decision Support: A Heuristic Evaluation of Explainable AI in Healthcare Dashboards

    B. V. Van Dort, J. Cañizares, R. Marcilly, et al. · 2025 · Studies in health technology and informatics · 2 citations

    Explainable Artificial Intelligence (XAI) is crucial for enhancing transparency, interpretability and actionability of AI systems, particularly in healthcare. The SAD XAI Dashboard, a clinical decision support (CDS) tool for sepsis-associated delirium (SAD), assists physicians in understanding AI-driven predictions for SAD. Our study aimed to evaluate the XAI dashboard's compliance with an XAI usability heuristics checklist. Three experts (human factors and health informatics) applied the heuristics checklist to the dashboard. The evaluation identified several usability issues, including unclear sequences of actions, non-standard icons, and inconsistent labelling. Key trust and transparency

    Save this paper

  21. Explainable AI for Clinical Decision Support: SHAP and Grad-CAM in Text- and Image-Based Medical Predictions

    Rohit Kshirsagar, Rishabh Kothari, Parth Lhase, et al. · 2025 · International Journal For Multidisciplinary Research · 2 citations

    The integration of Artifcial Intelligence (AI), par- ticularly deep learning, into clinical workfows promises to revo- lutionize medical diagnostics. High-performance models can now analyze medical images and clinical text with accuracy rivaling, and sometimes exceeding, human experts. However, the “black- box” nature of these complex models is a signifcant barrier to their widespread clinical adoption. Physicians require transpar- ent and interpretable reasoning to trust AI-driven predictions for high-stakes decisions. This survey addresses this critical need by providing a focused review of Explainable AI (XAI) techniques tailored for medical applications. We move beyond general discussion

    Save this paper

  22. A User-Centered Design of Explainable AI for Clinical Decision Support

    Mozhgan Salimiparsa, Daniel J. Lizotte, Kamran Sedig · 2021 · Proceedings of the Canadian Conference on Artificial Intelligence · 2 citations

    Clinical decision support (CDS) systems are computer applications whose goal is to facilitate the decision-making process of clinicians. In recent years, CDSS has developed an interest in applying machine learning (ML) models to make predictions related to clinical outcomes. The limited interpretability of many ML models is a major barrier to clinical adoption. This challenge has sparked research interest in interpretable and explainable AI, commonly known as XAI. XAI methods are used to construct and communicate explanations of the predictions made by machine learning models so that end users can interpret those predictions. However, these methods are not designed based on end-users' needs;

    Save this paper

  23. Explainable AI-Based Semantic Retrieval from an Expert-Curated Oncology Knowledge Graph for Clinical Decision Support

    S. Mushtaq, M. Trovati, N. Bessis · 2025 · Future Internet · 1 citations

    The modern oncology landscape is characterised by a deluge of high-dimensional data from genomic sequencing, medical imaging, and electronic health records, negatively impacting the analytical capacity of clinicians and health practitioners. This field is not new and it has drawn significant attention from the research community. However, one of the main limiting issues is the data itself. Despite the vast amount of available data, most of it lacks scalability, quality, and semantic information. This work is motivated by the data platform provided by OncoProAI, an AI-driven clinical decision support platform designed to address this challenge by enabling highly personalised, precision cancer

    Save this paper

Write your paper with these sources

Folio is the integrity-first research workspace: search 200M+ papers, save sources, and write with citations that format themselves. Free for students and researchers.

Start writing free →