LLM-Enhanced Anomaly Classification for Images
Classifying anomalies in images, that is assigning meaningful categories or descriptions to unexpected patterns, can automate quality control and enhance medical diagnostics. In this blog post, we explore how to leverage EfficientAD, an anomaly classification model for image data, and Large Language Models (LLMs) to perform anomaly classification on images in MATLAB.
The workflow presented not only automates anomaly classification for image data, but with LLM assistance, it also generates natural-language explanations about the classification results. I will also provide code snippets from a GitHub repository with the code you need for LLM-enhanced anomaly classification.
File Exchange Repository: Large Language Models (LLMs) with MATLAB

Key Concepts
What is Anomaly Classification?
Anomaly classification is the task of assigning meaningful labels or categories to patterns in image data that deviate from expected norms. Unlike general image classification, which identifies what is present in an image, anomaly classification focuses specifically on irregularities and determines what type of anomaly they represent. For example, in manufacturing, anomaly classification might involve distinguishing between a scratch, dent, or coating defect on a product surface. In medical imaging, it could mean classifying a lesion as benign or malignant. What sets anomaly classification apart is that the anomalies are often rare, diverse, and context dependent. This makes the classification more complex than traditional classification tasks. An effective anomaly classification model must not only recognize that an irregularity exists but also understand its nature, interpret its severity, and, in some cases, suggest appropriate responses. This requires models that are not just accurate, but also explainable and adaptable, especially in environments where new or previously unseen anomaly types can emerge over time.EfficientAD: A Model for Real-Time Anomaly Classification
EfficientAD is a high-performance visual anomaly classification model designed for real-time industrial applications. It uses a student-teacher architecture, where a lightweight student network is trained to mimic the feature representations of a larger, pre-trained teacher network, specifically on images that represent the normal or expected state. During inference, differences between the student and teacher outputs are used to classify anomalous patterns. This architecture is not only accurate but also very fast, capable of processing images in under 2 milliseconds and achieving throughput of over 600 images per second on modern GPUs. A key advantage of EfficientAD is its ability to handle both structural and logical anomalies. Structural anomalies, such as surface scratches or misalignments, are identified based on local feature deviations. Logical anomalies, on the other hand, involve inconsistencies in semantic structure, like unexpected object configurations. To support this, EfficientAD incorporates an autoencoder module that captures global image semantics, enhancing its classification capabilities beyond low-level appearance differences. The result is a fast, flexible model that delivers reliable performance even in compute-limited environments, making it well-suited for real-world applications. Since R2024b, EfficientAD is available in MATLAB by using the efficientADAnomalyDetector object. EfficientAD is included in Automated Visual Inspection Library for Computer Vision Toolbox, which offers functions for training, evaluating, and deploying anomaly classification, anomaly detection, and object detection networks.Enhancing Anomaly Classification with LLMs
While EfficientAD classifies visual anomalies, LLMs can contextualize the findings, offer domain-specific explanations, and enhance the explainability of results. In this context, LLMs help to:- Convert anomaly maps into human-readable insights.
- Suggest potential causes based on historical anomaly descriptions.
- Assist with automated anomaly reporting and recommendations.

Get the Anomaly Classification Code
My colleague Takuji Fukumoto has created a GitHub repository with code for performing anomaly classification. You can find the repo here: Zero-shot anomaly classification with EfficientAD and LLM. You may choose to jump right into it. However, if you keep reading, I will show you the key steps from the repository.Load Dataset
Get the image data, on which you will perform anomaly classification, and create an image datastore.dataDir = "CAEdemo"; downloadData(dataDir) Imgsize = [720 1280]; % Image size BlockSize = [256 256]; % Input size of Network numPatch =8; imds = imageDatastore(fullfile(dataDir,dataDir,"trainingimage"),IncludeSubfolders=true); imdsTrain = subset(imds,1:38); imdsTraint = transform(imdsTrain,@(x1) transformFcn(x1,BlockSize,numPatch));Display some sample images from the training dataset. You can see that they are fabric images and at first glance it’s very hard to detect any possible anomalies.
numTrainImages = numel(imdsTrain.Files); idx = randperm(numTrainImages,16); I = imtile(imdsTrain,Frames=idx,GridSize=[4 4],BorderSize=10); figure imshow(I)

Fine-Tune Anomaly Classifier
Create the EfficientAD anomaly classifier using the efficientADAnomalyDetector object. Specify the UseGlobalAnomalyMap argument as false to turn off the autoencoder and use only the student-teacher model.untrainedDetector = efficientADAnomalyDetector(Network="pdn-medium",UseGlobalAnomalyMap=false);Specify the training options and train the anomaly detector.
maximumEpochs=100; options = trainingOptions("adam", ... InitialLearnRate=1e-4, ... L2Regularization=1e-5, ... LearnRateSchedule="piecewise", ... LearnRateDropPeriod=floor(0.95*maximumEpochs), ... LearnRateDropFactor=0.1, ... MaxEpochs=maximumEpochs, ... VerboseFrequency=2, ... MiniBatchSize=1, ... Shuffle="every-epoch", ... ResetInputNormalization=true,... Plots="training-progress"); detector = trainEfficientADAnomalyDetector(imdsTraint,untrainedDetector,options,MapNormalizationDataRatio=0.2);Now, the anomaly classifier is ready to use for the task at hand.
Enhance Classification with LLM
The LLM that I am using here is a GPT model. To interface the ChatGPT API, you must obtain an OpenAI API key. To learn more about how to obtain the API key and charges for using the OpenAI API, see OpenAI API. Initialize the OpenAI Chat object as an AI assistant.chat = openAIChat("You are an AI assistant.",ApiKey=my_key);Provide context and images with anomaly maps, which are created by the EfficientAD anomaly detector. Plot the images with an explanation on the anomaly classification results in natural language, as generated by the LLM.
alltext = []; imdsllm = imageDatastore("cropped_image",IncludeSubfolders=true); for k =1:numel(imdsllm.Files) % Create context and query prompts messages = messageHistory; context = "You are inspecting the blue polka dot fabric as shown in this image file. This file is a normal data." + ... "It has a tested image on its left and a mask image showing where there is abnormal part on its right" + ... "If the test image is normal, the right side is completely black and no white area."; messages = addUserMessageWithImages(messages,context,string(imdsllm.Files{6})); image_path = imdsllm.Files{k}; Query = "This image is the one that was determined to be abnormal." + ... "It has a tested image on its left and a mask image showing where there is abnormal part on its right" + ... "Referencing white area of the mask image describe What anomalies the image on your left contains with in 10 words" + ... "It is possible that a foreign substance is mixed in, something different in color exist or there is some kind of stain on the fabric."+ ... "If the corresponding mask on your right is black and not white, There are no stains and foreign substances. Answer 'This is normal'"; messages = addUserMessageWithImages(messages,Query,string(image_path)); % Generate a response [txt,response] = generate(chat,messages,MaxNumTokens=4096,TimeOut=500); % Plot anomaly classification results enhanced by LLM figure(f), subplot(6,1,k), imshow(imoutall{k}); title(txt,'FontSize',10);shg end

Key Takeaways
- The EfficientAD model offers fast, accurate, and scalable anomaly classification by combining lightweight architecture with strong semantic understanding, which makes it ideal for real-time industrial applications.
- LLMs transform anomaly classification results into natural-language explanations, suggest possible causes based on historical data, and assist with automated anomaly reporting, making the classification results more interpretable and actionable.
- MATLAB provides an environment for integrating multiple AI and data preprocessing techniques, allowing you to build an end-to-end anomaly classification workflow. This unifying approach can simplify development, deployment, and explainability.
- Category:
- AI Application,
- Deep Learning
Comments
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.