{"id":11970,"date":"2023-05-08T08:49:41","date_gmt":"2023-05-08T12:49:41","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=11970"},"modified":"2024-05-29T12:09:12","modified_gmt":"2024-05-29T16:09:12","slug":"explainable-ai-xai-are-we-there-yet","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2023\/05\/08\/explainable-ai-xai-are-we-there-yet\/","title":{"rendered":"Explainable AI (XAI): Are we there yet?"},"content":{"rendered":"
<\/h6>\r\nThis post is from Oge Marques<\/a>, PhD and Professor of Engineering and Computer Science at FAU.<\/em>\r\n
<\/h6>\r\n \r\n
<\/h6>\r\n
This is the second post in a 3-post series on Explainable AI<\/strong> (XAI). The first post<\/a> highlighted examples and offered practical advice on how and when to use XAI techniques for computer vision tasks. In this post, we will expand the discussion by offering advice on some of the XAI limitations and challenges that you might find along the path of XAI adoption.<\/blockquote>\r\n
<\/h6>\r\nAt this point, you might have heard about Explainable AI and want to explore XAI techniques within your work. But along the way you might also have wondered : Just how well do XAI techniques work? And what should you do if you find the results of XAI aren\u2019t as compelling as you\u2019d like? Below, we explore some of the questions surrounding XAI with a healthy dose of skepticism.\r\n
<\/h6>\r\n \r\n

Are expectations too high?<\/strong><\/p>\r\nXAI \u2013 like many subfields of AI \u2013 can be subject to some of the hype, fallacies, and misplaced hopes associated with AI at large, including the potential for unrealistic expectations.\r\n

<\/h6>\r\nTerms such as explainable<\/em> and explanation<\/em> risk falling under the fallacy of matching AI developments with human abilities (the 2021 paper Why AI is Harder Thank We Think<\/a>\u00a0brilliantly describes four similar fallacies in AI assumptions). This fallacy leads to the unrealistic expectation that we are approaching a point in time where AI solutions will not only achieve great feats and eventually surpass human intelligence, but \u2013 on top of that \u2013 they will be able to explain how<\/em> and why<\/em> AI did what it did; therefore, increasing the level of trust in AI decisions. Once we become aware of this fallacy, it is legitimate to ask ourselves: How much can we realistically<\/strong> expect from XAI?<\/em>\r\n
<\/h6>\r\nIn this post we will discuss whether our expectations for XAI (and the value XAI techniques might add to our work) are too high and focus on answering three main questions:\r\n
<\/h6>\r\n
    \r\n \t
  1. Can XAI become a proxy for trust(worthiness)?<\/a><\/li>\r\n \t
  2. What limitations of XAI techniques should we be aware of?<\/a><\/li>\r\n \t
  3. Can consistency across XAI techniques improve user experience?<\/a><\/li>\r\n<\/ol>\r\n
    <\/h6>\r\nLet\u2019s look at these three specific aspects of XAI-related expectations in more detail.\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\n

    <\/a>1. Can XAI become a proxy for trust(worthiness)?<\/strong><\/p>\r\nIt is natural to associate an AI solution\u2019s \u201cability to explain itself\u201d to the degree of trust humans place in that solution. However, should explainability translate to trustworthiness?\r\n

    <\/h6>\r\nXAI, despite its usefulness, falls short of ensuring trust in AI for at least these three reasons:\r\n
    <\/h6>\r\nA.\u00a0 <\/strong> <\/span>The premise that AI algorithms must be able to explain their decisions to humans can be problematic. Humans cannot always explain their decision-making process, and even when they do, their explanations might not be reliable or consistent.\r\n
    <\/h6>\r\nB.\u00a0 <\/strong> <\/span> Having complete trust in AI involves trusting not only the model, but also the training data, the team that created the AI model, and the entire software ecosystem. To ensure parity between models and outcomes, best practices and conventions for data standards, workflow standards, and technical subject matter expertise must be followed.\r\n
    <\/h6>\r\nC.\u00a0 <\/strong> <\/span> Trust is something that develops over time, but in the age where early adopters want models as fast as they can be produced, models are sometimes introduced to the public early with the expectation they will be continuously updated and improved upon in later releases \u2013 which might not happen. In many applications, adopting AI models that stand the test of time by showing consistently reliable performance and correct results over time might be good enough, even without any XAI capabilities.\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\n

    <\/a>2. What limitations of XAI techniques should we be aware of?<\/strong><\/p>\r\nIn the fields of image analysis and computer vision, a common interface for showing the results of XAI techniques consists of overlaying the \u201cexplanation\u201d (usually in the form of a\u00a0heatmap\u00a0or\u00a0saliency map) on top of the image. This can be helpful in determining which areas of the image the model deemed to be most relevant in its decision-making process. It can also assist in diagnosing potential blunders that the deep learning model might be making, which produce results that are seemingly correct but in reality the model was looking in wrong place. A classic example is the husky vs. wolf<\/em>\u00a0image classification algorithm<\/a>\u00a0which in fact was a snow detector (Figure 1).\r\n\r\n \r\n

    \"Husky<\/h6>\r\nFigure 1<\/strong>: Example of misclassification in a \u201chusky vs. wolf\u201d image classifier due to a spurious correlation between images of wolves and the presence of snow. \u00a0The image on the right, which shows the result of the LIME post-hoc XAI technique , captures the classifier blunder. Source: https:\/\/arxiv.org\/pdf\/1602.04938.pdf<\/a>\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\nThe potential usefulness of post-hoc XAI results has led to a growing adoption of techniques, such as imageLIME<\/a>, occlusion sensitivity<\/a>, gradCAM<\/a> for computer vision tasks. However, these XAI techniques sometimes fall short of delivering the desired explanation due to some well-known limitations. Below are 3 examples:\r\n
    <\/h6>\r\n

    \u2003\u2003A.\u00a0 <\/strong> <\/span> XAI techniques can sometimes use similar explanations for correct and incorrect decisions.<\/p>\r\nFor example, in the previous post in this series<\/a>, we showed that the heatmaps produced by the gradCAM function provided similarly convincing explanations (in this case, focus on the head area of the dog) both when their prediction is correct (Figure 2, top) as well as incorrect (Figure 2, bottom: a Labrador retriever was mistakenly identified as a beagle).\r\n

    <\/h6>\r\n \r\n
    <\/h6>\r\n
    \"Classify<\/h6>\r\n
    <\/h6>\r\nFigure 2<\/strong>: Example of results using gradCAM for the dog breed image classification task.\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\n

    \u2003\u2003B.\u00a0 <\/strong> <\/span> Even in cases where XAI techniques show that a model is not looking in the right place, that doesn\u2019t necessarily mean that it is easy to know how to fix the underlying problem.<\/p>\r\nIn some easier cases, such as in the husky vs. wolf classification mentioned earlier, a quick visual inspection of model errors could have helped identify the spurious correlation between \u201cpresence of snow\u201d and \u201cimages of wolves.\u201d There is no guarantee that the same process would work for other (larger or more complex) tasks.\r\n

    <\/h6>\r\n

    \u2003\u2003C.\u00a0 <\/strong> <\/span> Results are model- and task-dependent.<\/p>\r\nIn our previous post in this\u00a0 series, we also showed that the heatmaps produced by the gradCAM function in MATLAB provided different visual explanations associated with the same image (Figure 3) depending on the pretrained model and task. Figure 4 shows those two examples and adds a third example<\/a>, in which the same network (GoogLeNet) was used without modification. A quick visual inspection of Figure 4 is enough to spot significant differences among the three heatmaps.\r\n

    <\/h6>\r\n\"Labrador\r\n
    <\/h6>\r\nFigure 3<\/strong>: Test image for different image classification tasks and models (shown in Figure 4).\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\n\"Classification\r\n
    <\/h6>\r\nFigure 4<\/strong>: Using gradCAM on the same test image (Figure 3), but for different image classification tasks and models.\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\n

    <\/a>3. \u00a0Can consistency across XAI techniques improve user experience?<\/strong><\/p>\r\nIn order to understand the\u00a0why<\/em>\u00a0and\u00a0how<\/em>\u00a0behind an AI model\u2019s decisions and get a better insight into its successes and failures, an explainable model should be capable of explaining itself to a human user through some type of\u00a0explanation interface\u00a0(Figure 5). Ideally this interface should be rich, interactive, intuitive, and appropriate for the user and task.\r\n

    <\/h6>\r\n\"User\r\n
    <\/h6>\r\nFigure 5<\/strong>: Usability (UI\/UX) aspects of XAI: an explainable model requires a suitable interface.\r\n
    <\/h6>\r\n \r\n
    <\/h6>\r\nIn visual AI tasks, comparing the results produced by two or more methods<\/a> can be problematic, since the most commonly used post-hoc XAI methods use significantly different explanation interfaces (that is, visualization schemes) in their implementation (Figure 6):\r\n
    <\/h6>\r\n