Classifying anomalies in images, that is assigning meaningful categories or descriptions to unexpected patterns, can automate quality control and enhance medical diagnostics. In this blog post, we explore how to leverage EfficientAD, an anomaly classification model for image data, and Large Language Models (LLMs) to perform anomaly classification on images in MATLAB.

The workflow presented not only automates anomaly classification for image data, but with LLM assistance, it also generates natural-language explanations about the classification results. I will also provide code snippets from a GitHub repository with the code you need for LLM-enhanced anomaly classification.

 

Key Concepts

What is Anomaly Classification?

Anomaly classification is the task of assigning meaningful labels or categories to patterns in image data that deviate from expected norms. Unlike general image classification, which identifies what is present in an image, anomaly classification focuses specifically on irregularities and determines what type of anomaly they represent. For example, in manufacturing, anomaly classification might involve distinguishing between a scratch, dent, or coating defect on a product surface. In medical imaging, it could mean classifying a lesion as benign or malignant.

What sets anomaly classification apart is that the anomalies are often rare, diverse, and context dependent. This makes the classification more complex than traditional classification tasks. An effective anomaly classification model must not only recognize that an irregularity exists but also understand its nature, interpret its severity, and, in some cases, suggest appropriate responses. This requires models that are not just accurate, but also explainable and adaptable, especially in environments where new or previously unseen anomaly types can emerge over time.

EfficientAD: A Model for Real-Time Anomaly Classification

EfficientAD is a high-performance visual anomaly classification model designed for real-time industrial applications. It uses a student-teacher architecture, where a lightweight student network is trained to mimic the feature representations of a larger, pre-trained teacher network, specifically on images that represent the normal or expected state. During inference, differences between the student and teacher outputs are used to classify anomalous patterns. This architecture is not only accurate but also very fast, capable of processing images in under 2 milliseconds and achieving throughput of over 600 images per second on modern GPUs.

A key advantage of EfficientAD is its ability to handle both structural and logical anomalies. Structural anomalies, such as surface scratches or misalignments, are identified based on local feature deviations. Logical anomalies, on the other hand, involve inconsistencies in semantic structure, like unexpected object configurations. To support this, EfficientAD incorporates an autoencoder module that captures global image semantics, enhancing its classification capabilities beyond low-level appearance differences. The result is a fast, flexible model that delivers reliable performance even in compute-limited environments, making it well-suited for real-world applications.

Since R2024b, EfficientAD is available in MATLAB by using the efficientADAnomalyDetector object. EfficientAD is included in Automated Visual Inspection Library for Computer Vision Toolbox, which offers functions for training, evaluating, and deploying anomaly classification, anomaly detection, and object detection networks.

Enhancing Anomaly Classification with LLMs

While EfficientAD classifies visual anomalies, LLMs can contextualize the findings, offer domain-specific explanations, and enhance the explainability of results. In this context, LLMs help to:

• Convert anomaly maps into human-readable insights.
• Suggest potential causes based on historical anomaly descriptions.
• Assist with automated anomaly reporting and recommendations.

You can access popular LLMs, such as models, from MATLAB through an API or by installing the models locally. Then, you can use your preferred model to analyze and generate text. The code you need to access and interact with LLMs using MATLAB is in the Large Language (LLMs) with MATLAB repository.

You have three options for accessing LLMs. You can connect MATLAB to the OpenAI® Chat Completions API (which powers ChatGPT™), Ollama™ (for local LLMs), and Azure® OpenAI services. To learn more about these options, check out these previous blog posts: blog post: OpenAI LLMs with MATLAB and blog post: Local LLMs with MATLAB.

File Exchange Repository: Large Language Models (LLMs) with MATLAB

 

Get the Anomaly Classification Code

My colleague Takuji Fukumoto has created a GitHub repository with code for performing anomaly classification. You can find the repo here: Zero-shot anomaly classification with EfficientAD and LLM. You may choose to jump right into it. However, if you keep reading, I will show you the key steps from the repository.

Load Dataset

Get the image data, on which you will perform anomaly classification, and create an image datastore.

dataDir = "CAEdemo";
downloadData(dataDir)

Imgsize = [720 1280]; % Image size
BlockSize = [256 256]; % Input size of Network
numPatch =8;

imds = imageDatastore(fullfile(dataDir,dataDir,"trainingimage"),IncludeSubfolders=true);
imdsTrain = subset(imds,1:38);
imdsTraint = transform(imdsTrain,@(x1) transformFcn(x1,BlockSize,numPatch));

Display some sample images from the training dataset. You can see that they are fabric images and at first glance it’s very hard to detect any possible anomalies.

numTrainImages = numel(imdsTrain.Files);
idx = randperm(numTrainImages,16);
I = imtile(imdsTrain,Frames=idx,GridSize=[4 4],BorderSize=10);
figure
imshow(I)

 

Fine-Tune Anomaly Classifier

Create the EfficientAD anomaly classifier using the efficientADAnomalyDetector object. Specify the UseGlobalAnomalyMap argument as false to turn off the autoencoder and use only the student-teacher model.

untrainedDetector = efficientADAnomalyDetector(Network="pdn-medium",UseGlobalAnomalyMap=false);

Specify the training options and train the anomaly detector.

maximumEpochs=100;
options = trainingOptions("adam", ...
    InitialLearnRate=1e-4, ...
    L2Regularization=1e-5, ...
    LearnRateSchedule="piecewise", ...
    LearnRateDropPeriod=floor(0.95*maximumEpochs), ...
    LearnRateDropFactor=0.1, ...
    MaxEpochs=maximumEpochs, ...
    VerboseFrequency=2, ...
    MiniBatchSize=1, ...
    Shuffle="every-epoch", ...
    ResetInputNormalization=true,...
    Plots="training-progress");

detector = trainEfficientADAnomalyDetector(imdsTraint,untrainedDetector,options,MapNormalizationDataRatio=0.2);

Now, the anomaly classifier is ready to use for the task at hand.

Enhance Classification with LLM

The LLM that I am using here is a GPT model. To interface the ChatGPT API, you must obtain an OpenAI API key. To learn more about how to obtain the API key and charges for using the OpenAI API, see OpenAI API.

Initialize the OpenAI Chat object as an AI assistant.

chat = openAIChat("You are an AI assistant.",ApiKey=my_key);

Provide context and images with anomaly maps, which are created by the EfficientAD anomaly detector. Plot the images with an explanation on the anomaly classification results in natural language, as generated by the LLM.

alltext = [];
imdsllm = imageDatastore("cropped_image",IncludeSubfolders=true);

for k =1:numel(imdsllm.Files)
    % Create context and query prompts
    messages = messageHistory;
    context = "You are inspecting the blue polka dot fabric as shown in this image file. This file is a normal data." + ...
        "It has a tested image on its left and a mask image showing where there is abnormal part on its right" + ...
        "If the test image is normal, the right side is completely black and no white area.";
    messages = addUserMessageWithImages(messages,context,string(imdsllm.Files{6}));

    image_path = imdsllm.Files{k};
    Query = "This image is the one that was determined to be abnormal." + ...
        "It has a tested image on its left and a mask image showing where there is abnormal part on its right" + ...
        "Referencing white area of the mask image describe What anomalies the image on your left contains with in 10 words" + ...
        "It is possible that a foreign substance is mixed in, something different in color exist or there is some kind of stain on the fabric."+ ...
        "If the corresponding mask on your right is black and not white, There are no stains and foreign substances. Answer 'This is normal'";
    messages = addUserMessageWithImages(messages,Query,string(image_path));
    
    % Generate a response
    [txt,response] = generate(chat,messages,MaxNumTokens=4096,TimeOut=500);
    
    % Plot anomaly classification results enhanced by LLM
    figure(f), subplot(6,1,k), imshow(imoutall{k});
    title(txt,'FontSize',10);shg
end

 

Key Takeaways

1. The EfficientAD model offers fast, accurate, and scalable anomaly classification by combining lightweight architecture with strong semantic understanding, which makes it ideal for real-time industrial applications.
2. LLMs transform anomaly classification results into natural-language explanations, suggest possible causes based on historical data, and assist with automated anomaly reporting, making the classification results more interpretable and actionable.
3. MATLAB provides an environment for integrating multiple AI and data preprocessing techniques, allowing you to build an end-to-end anomaly classification workflow. This unifying approach can simplify development, deployment, and explainability.