{"id":12278,"date":"2023-06-27T10:09:16","date_gmt":"2023-06-27T14:09:16","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=12278"},"modified":"2024-05-29T12:17:36","modified_gmt":"2024-05-29T16:17:36","slug":"explainable-ai-xai-implement-explainability-in-your-work","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2023\/06\/27\/explainable-ai-xai-implement-explainability-in-your-work\/","title":{"rendered":"Explainable AI (XAI): Implement explainability in your work"},"content":{"rendered":"
<\/h6>\r\nThis post is from Oge Marques<\/a>, PhD and Professor of Engineering and Computer Science at FAU.<\/em>\r\n
<\/h6>\r\n
This is the third post in a 3-post series on Explainable AI<\/strong> (XAI)<\/em>. In the first post<\/a>, we showed examples and offered practical advice on how and when to use \u00a0XAI techniques for computer vision tasks. In the second post<\/a>, we offered words of caution and discussed the limitations. In this post, we conclude the series by offering a practical guide for getting started with explainability, including tips and examples.<\/blockquote>\r\n
<\/h6>\r\nIn this blog post, we focus on image classification tasks and offer 4 practical tips, which help you make the most of Explainable AI techniques, for those of you ready to implement explainability in your work.\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

TIP 1: Why is explainability important?<\/strong><\/p>\r\nBefore you dive into the numerous practical details related to using XAI techniques in your work, you should start by examining your reasons for using explainability. Explainability can help you better understand your model\u2019s predictions and reveal inaccuracies in your model and bias in your data.\r\n

<\/h6>\r\nIn the second blog post of this series, we commented on the use of post-hoc XAI techniques to assist in diagnosing potential blunders that the deep learning model might be making; that is, producing results that are seemingly correct but reveal that the model was \u201clooking at the wrong places.\u201d A classic example in the literature demonstrated that a husky vs. wolf<\/em>\u00a0image classification algorithm<\/a>\u00a0was, in fact, a \u201csnow detector.\u201d (Fig. 1).\r\n
<\/h6>\r\n\"Explainability\r\n
<\/h6>\r\nFigure 1:<\/strong> Example of misclassification in a \u201chusky vs. wolf\u201d image classifier due to a spurious correlation between images of wolves and the presence of snow.\u00a0 The image on the right, which shows the result of the LIME post-hoc XAI technique, captures the classifier blunder. [Source<\/a>]\r\n
<\/h6>\r\nThese are examples where there is not much at stake. But what about high-stakes areas (such as healthcare) and sensitive topics in AI (such as bias and fairness)? In the field of radiology, there is a famous example<\/a> where models designed to identify pneumonia in chest X-rays learned to recognize a metallic marker placed by radiology technicians in the corner of the image (Fig. 2). This marker is typically used to indicate the source hospital where the image was taken. As a result, the models performed effectively when analyzing images from the hospital they were trained on, but struggled when presented with images from other hospitals that had different markers. And most importantly, explainable AI revealed that the models were not diagnosing pneumonia but classifying the presence of metallic markers.\r\n
<\/h6>\r\n\"Explainable\r\n
<\/h6>\r\nFigure 2:<\/strong> A deep learning model for detecting pneumonia: the CNN has learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image. When these strong features are correlated with disease prevalence, models can leverage them to indirectly predict disease. [Source<\/a>]\r\n
<\/h6>\r\nExample<\/strong><\/span>\r\n
<\/h6>\r\nThis example<\/a> shows MATLAB code to produce post-hoc explanations (using two popular post-hoc XAI techniques,\u00a0Grad-CAM<\/a>\u00a0and\u00a0image LIME<\/a>) for a medical image classification task.\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

TIP 2: Can you use an inherently explainable model?<\/strong><\/p>\r\nDeep learning models are often the first choice to consider, but should they be? For problems involving (alphanumerical) tabular data there are numerous interpretable ML techniques to choose from, including: decision trees, linear regression, logistic regression, Generalized Linear Models (GLMs), and Generalized Additive Models (GAMs). In computer vision, however, the prevalence of deep learning architectures such as convolutional neural networks (CNNs) and, more recently, vision transformers, make it necessary to implement mechanisms for visualizing network predictions after the fact.\r\n

<\/h6>\r\nIn a landmark paper<\/a>, Duke University researcher and professor Cynthia Rudin made a strong claim in favor of the interpretable models (rather than post-hoc XAI techniques applied to an opaque model). Alas, prescribing the use of interpretable models and successfully are two dramatically different things; for example, an interpretable model from Rudin\u2019s research group, ProtoPNet<\/a>, has achieved relatively modest success and popularity.\r\n
<\/h6>\r\nIn summary, from a pragmatic standpoint, you are better off using pretrained models such as the ones available here<\/a> and dealing with their opaqueness through judicious use of post-hoc XAI techniques than embarking on a time-consuming research project.\r\n
<\/h6>\r\nExample<\/strong><\/span>\r\n
<\/h6>\r\nThis MATLAB page<\/a> provides a brief overview of interpretability and explainability, with links to many code examples.\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

TIP 3: How to choose the right explainability technique?<\/strong><\/p>\r\nThere are many post-hoc XAI techniques to choose from \u2013 and several of them have become available as MATLAB library functions, including Grad-CAM<\/a> and LIME<\/a>. These are two of the most popular methods in an ever-growing field that has more than 30 techniques to choose from (as of Dec 2022<\/a>). Consequently, selecting the best method can be intimidating at first. As with many other decisions in AI, I advise to start with the most popular, broadly available methods first. Later, if you collect enough evidence (for example by running experiments with users of the AI solution) that certain techniques work best in certain contexts, you can test and adopt other methods.\r\n

<\/h6>\r\nIn the case of image classification, the perception of added value provided by the XAI technique can also be associated to the visual display of results. Fig. 3 provides five examples of visualization of XAI results using different techniques. The visual results are significantly different among them, which might lead to different users preferring different methods.\r\n
<\/h6>\r\n\"Deep\r\n
<\/h6>\r\nFigure 3:<\/strong> Examples of different post-hoc XAI techniques and associated visualization options. [Source<\/a>]\r\n
<\/h6>\r\nExample<\/strong><\/span>\r\n
<\/h6>\r\nThe GUI-based UNPIC app<\/a> allows you to explore the predictions of an image classification model using several deep learning visualization and XAI techniques.\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

TIP 4: Can you improve XAI results and make them more user-centric?<\/strong><\/p>\r\nYou can view explainable AI techniques as one option for interpreting the model\u2019s decisions along with a range of other options (Fig. 4). For example, in medical image classification, an AI solution that predicts a medical condition from a patient\u2019s chest x-ray might use gradually increasing degrees of explainability: (1) no explainability information<\/strong>, just the outcome\/prediction; (2) adding output probabilities<\/strong> for most likely predictions, giving a measure of confidence associated with them; (3) adding visual saliency<\/strong> information describing areas of the image driving the prediction; (4) combining predictions with results from a medical case retrieval (MCR) system and indicating matched real cases<\/strong> that could have influenced the prediction; and (5) adding computer-generated semantic explanation<\/strong>.\r\n

<\/h6>\r\n\"Explainability\r\n
<\/h6>\r\nFigure 4: <\/strong>XAI as a gradual approach: in addition to the model\u2019s prediction, different types of supporting information can be added to explain the decision. [Source<\/a>]\r\n
<\/h6>\r\nExample<\/strong><\/span>\r\n
<\/h6>\r\nThis example<\/a> shows MATLAB code to produce post-hoc explanations (heat maps) and output probabilities for a food image classification task and demonstrates their usefulness in the evaluation of misclassification results.\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

A cheat sheet with practical suggestions, tips, and tricks<\/strong><\/p>\r\nUsing post-hoc XAI can help but it shouldn\u2019t be seen as a panacea. We hope the discussions, ideas, and suggestions in this blog series were useful to your professional needs. To conclude, we present a cheat sheet with some key suggestions for those who want to employ explainable AI in their work:\r\n

<\/h6>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
1<\/strong><\/span><\/td>\r\nStart with a clear understanding of the problem you are trying to solve and the specific reasons why you might want to use explainable AI models.<\/span><\/td>\r\n<\/tr>\r\n
<\/td>\r\n\"\"<\/td>\r\n<\/tr>\r\n
2<\/strong><\/span><\/td>\r\nWhenever possible, use an inherently explainable model, \u00a0such as a decision tree, or a rule-based model. On the other hand, CNNs are amenable to post-hoc XAI techniques based on gradients and weight values.<\/span><\/td>\r\n<\/tr>\r\n
<\/td>\r\n\"\"<\/td>\r\n<\/tr>\r\n
3<\/strong><\/span><\/td>\r\nVisualize and assess the model outputs. This can help you understand how the model is making decisions and identify any issues that may arise.<\/span><\/td>\r\n<\/tr>\r\n
<\/td>\r\n\"\"<\/td>\r\n<\/tr>\r\n
4<\/strong><\/span><\/td>\r\nConsider providing additional context around the decision-making process to end-users, such as feature importance or sensitivity analysis. This can help build trust in the model and increase transparency.<\/span><\/td>\r\n<\/tr>\r\n
<\/td>\r\n\"\"<\/td>\r\n<\/tr>\r\n
5<\/strong><\/span><\/td>\r\nFinally, document the entire process, including the data used, the model architecture, and the methods used to evaluate the model's performance. This will ensure reproducibility and allow others to verify your results.<\/span><\/td>\r\n<\/tr>\r\n
<\/td>\r\n\"\"<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n

Read more about it:<\/strong><\/p>\r\n\r\n