Artificial Intelligence https://blogs.mathworks.com/deep-learning Johanna specializes in deep learning and computer vision. Her goal is to give insight into deep learning through code examples, developer Q&As, and tips and tricks using MATLAB. Mon, 11 Mar 2024 15:16:33 +0000 en-US hourly 1 https://wordpress.org/?v=6.2.2 Incremental Learning: Adaptive and real-time machine learning https://blogs.mathworks.com/deep-learning/2024/03/04/incremental-learning-adaptive-and-real-time-machine-learning/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2024/03/04/incremental-learning-adaptive-and-real-time-machine-learning/#respond Mon, 04 Mar 2024 14:36:33 +0000 https://blogs.mathworks.com/deep-learning/?p=14044

  Incremental learning is a machine learning approach that addresses the challenge of adaptively fitting models to new incoming data. The incremental learning approach is particularly... read more >>

]]>
 
Incremental learning is a machine learning approach that addresses the challenge of adaptively fitting models to new incoming data. The incremental learning approach is particularly useful to engineers that need to model streaming data. Often, engineers and other AI practitioners deploy machine learning to target devices, and incremental learning ensures that the models continue to work as intended if the data changes.
In this blog post, we are going to explain what incremental learning is, why it is useful, and how to implement incremental learning with MATLAB tools and Simulink blocks.
 

What is Incremental Learning?

Incremental learning is a machine learning approach that enables machine learning models (and deep learning models) to continuously learn by processing incoming non-stationary data from a data stream. With incremental learning, you can create artificial intelligence (AI) systems that continuously update to integrate new knowledge while maintaining previous knowledge.
Diagram of the incremental learning workflow showing how a machine learning model learns with streaming data while maintaining previous knowledge.
Figure: Incremental learning workflow.
 

Incremental Learning vs Traditional Machine Learning

A traditional machine learning model is trained on a batch of data and generalization to new data (that is, avoiding overfitting or underfitting) is ensured by methods like cross-validation, regularization, and hyperparameter tuning.
On the other hand, incremental learning adapts to new data in real time, and therefore it provides certain benefits compared to traditional machine learning. Incremental learning is flexible, quick, and adaptive to new data. An incremental learning model fits to data quickly and efficiently, which means it can adapt in real time to changes (or drifts) in the data distribution. It is also more efficient when little information is known about the training data. For example, class names might not be known until after the model processes observations.
Additionally, incremental learning has these benefits:
  • Protecting the privacy of end-user data.
  • Allowing devices to learn even with limited or no internet connectivity.
  • Allowing the design of advanced devices with personalization and smart features.

Challenges in Incremental Learning

Incremental learning is not without its inherent challenges, a couple of which are data storage and catastrophic forgetting.
Data storage – Data arrives in a stream and the sample size is unknown and possibly large, which makes data storage difficult. Therefore, the incremental learning algorithm must process the data when they are available and before they get discarded.
Catastrophic forgetting – An incremental learning model can’t access previous data while learning on new data. The model can overfit on the new data, which results in poor model performance.
 

Incremental Anomaly Detection

Incremental anomaly detection is a branch of machine learning that, similarly to incremental learning, involves processing incoming data from a data stream. In incremental anomaly detection, instead of fitting a machine learning model, the algorithm computes anomaly scores in real time.

Learn More About Incremental Learning

To learn more about what incremental learning is and get started with an example, see:
Showing the cumulative and windowed classification error decreasing for incremental learning.
Figure: Classification error for incremental learning model using flexible workflow by updating the performance metrics.
 

Why Is Incremental Learning Useful?

To solve real world problems, machine learning models must leave the desktop and go into production. When a machine learning model is operating on its target device, such as on the cloud or an edge device, the machine learning model is likely to receive non-stationary streaming data. This is when incremental learning is particularly useful.

Applications of Incremental Learning

Lithium-ion batteries are everywhere today, from wearable electronics, mobile phones, and laptops to electric vehicles and smart grids.  Let’s say you are designing a virtual sensor using AI to estimate the battery’s State-Of-Charge (SOC). An SOC virtual sensor is a key component of a battery management system (BMS) that ensures the safe and efficient operation of a battery. The virtual sensor receives voltage, current, and temperature measurements from other sensors. These measurements are likely to change over time and the model that you have deployed should adapt to these changes.
Diagram of a virtual sensor with inputs voltage, current, and temperature measurements, and output the State of Charge of a battery.
Figure: Designing a virtual sensor for battery State-Of-Charge (SOC) estimation using AI.
 
The design of virtual sensors is just one potential application of incremental learning. Other applications include:
Signal Processing Predictive Maintenance Wireless Communications
An example from my personal experience is using incremental learning in the design of implantable brain-machine interfaces (BMIs). During my PhD research, I developed algorithms and designed chips for implantable BMIs. The algorithms aimed to model very noisy brain signals and cluster brain activity to identify which neuron fired and when. Because all the preprocessing and machine learning must happen on an ultra-low power and tiny chip, the algorithms must be computationally efficient, have a small footprint, and process data in real time.
As part of my work, I developed an incremental learning algorithm that clustered the incoming neural signals in real time, while retaining information (like the cluster centers and statistical dependencies) of previously clustered activity. I wish ten years ago, MATLAB had built-in algorithms for incremental learning, but more on the recent tools available in MATLAB for incremental learning in the next section.

Incremental Learning and MLOps

MLOps is as a set of practices that automate the process of taking machine learning models to production, and managing the models once they are in production. As part of MLOps, machine learning models in production are constantly monitored. By using incremental learning algorithms, the machine learning models can be updated on-the-fly, which potentially reduces errors.
MLOps loop showing steps for machine learning and operations.
Figure: The MLOps lifecycle.
 

Consider that in real-world applications, data is often dynamic and always changing. So, drift can be a big issue for machine learning models. A data drift can happen for many reasons, such as changes in the distribution of the input data over time or the relationship between the input and desired output.

With incremental learning, the model is updated when the input changes.

         

Video: What is MLOps?

 

How to Implement Incremental Learning

Now that you understand what incremental learning is and how useful it is for modeling streaming data, we will describe MATLAB and Simulink tools so that you can easily implement incremental learning in your application.

Incremental Learning with MATLAB

Using algorithms from Statistics and Machine Learning Toolbox, you can create flexible, efficient, and adaptive incremental learning models for classification and regression, such as linear support vector (SVM), logistic regression, and naive Bayes classifiers, and least-squares and linear SVM regression models. Alternatively, you can convert a traditionally trained model to an incremental learning model by using the incrementalLearner function. To learn more about these incremental learning models, see the documentation topic Incremental Learning Overview.
With Statistics and Machine Learning Toolbox, you can detect concept drift for incremental learning models, that is, detect when the data has changed so that the model is no longer valid. Also, you can automatically generate C/C++ code for incremental learning models. To learn more, see the example Code Generation for Incremental Learning.
Graph with concept drift detection for incremental learning showing stable, warning, and drift status for different observations.
Figure: Concept drift detection for incremental learning with MATLAB.
 

Incremental Learning with Simulink

Using Simulink blocks provided in Statistics and Machine Learning Toolbox, you can integrate incremental learning into the design, simulation, and test of complex AI engineered systems, such as in the design of virtual sensors. To learn more, see the following examples:
Simulink system that includes the IncrementalClassificationLinear Fit and IncrementalClassificationLinear Predict blocks for incremental learning. Simulink system that includes the IncrementalRegressionLinear Fit and IncrementalRegressionLinear Predict blocks for incremental learning.
Incremental Learning in Simulink for Classification Incremental Learning in Simulink for Regression
 

Takeaways

  • Incremental learning addresses the challenge of fitting machine learning models adaptively to incoming streaming data.
  • Incremental learning can reduce errors when machine learning models are operating in production.
  • MATLAB and Simulink provide tools, functions, and blocks to create incremental learning models, integrate them into system-level design, and deploy them to hardware.
]]>
https://blogs.mathworks.com/deep-learning/2024/03/04/incremental-learning-adaptive-and-real-time-machine-learning/feed/ 0
Data-Driven Control with MATLAB and Simulink https://blogs.mathworks.com/deep-learning/2024/02/21/data-driven-control-with-matlab-and-simulink/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2024/02/21/data-driven-control-with-matlab-and-simulink/#respond Thu, 22 Feb 2024 01:46:40 +0000 https://blogs.mathworks.com/deep-learning/?p=13992

The following blog post is from Melda Ulusoy, Technical Marketing Manager at MathWorks. One of artificial intelligence (AI)’s first big successes was solving image classification problems with... read more >>

]]>
The following blog post is from Melda Ulusoy, Technical Marketing Manager at MathWorks.
One of artificial intelligence (AI)’s first big successes was solving image classification problems with deep learning. AI has since been used in many other areas, including control systems. In this blog post, we will present an overview of AI for controls, highlight advantages of using MATLAB and Simulink for data-driven control, and provide details on how to register for an upcoming webinar.
 

AI for Control Algorithms

Feedback control algorithms are used in advanced robots, electric motors, batteries, power converters, power grids, and autonomous vehicles that drive, fly, and sail. Traditionally, feedback control algorithms relied on linear models of the machines and devices, for which a control system engineer needed to develop a control algorithm.
As control engineers strive to improve the performance of control algorithms, they are increasingly turning to techniques that can enhance performance by considering the nonlinear dynamics of the systems to control.  AI techniques are great for creating accurate nonlinear models from data. Additionally, AI is very useful when control algorithms do not rely on a model of a system, but instead learn directly from data. So, control engineers are increasingly interested in applying AI to their work.
In this post, we cover a wide set of techniques for data-driven control. These techniques use system data to either learn a model of the system or directly learn control system parameters from data.  Some of the data-driven control techniques are AI-based algorithms, while others use non-AI-based algorithms to take advantage of system data.
 

Why MATLAB and Simulink for Data-Driven Control?

With MATLAB and Simulink, you can design and implement a variety of data-driven controllers including extremum seeking control (ESC), active disturbance rejection control (ADRC), model reference adaptive control (MRAC), data-driven model predictive control (MPC), and reinforcement learning (RL). This is not an exhaustive list, but rather a sample of somewhat recent capabilities that have been added to the data-driven control area.
Application areas of AI and data-driven control include ESC, ADRC, MRAC, MPC, and RL.
Figure: Sample of recent capabilities in the data-driven control area.
 
Using MATLAB and Simulink for data-driven control comes with several advantages. Engineers can:
  • Design, test, and compare a variety of control algorithms including both traditional and data-driven techniques in a single environment.
  • Implement and test data-driven control algorithms in Simulink using pre-built Simulink blocks.
  • Automatically generate code from the control algorithm for deployment to embedded hardware.
  • Use reference examples for flight control, robotics, energy management, and other applications to quickly get started with the implementation of data-driven control algorithms.

Watch Recent Webinar

This has been a short intro to data-driven control. For a deeper dive, watch the recent webinar on Data-Driven Control with MATLAB and Simulink.
In the webinar, you will learn the basics of ADRC, MPC, and RL, and see the following demonstrations:
  • Demo 1: Permanent magnet synchronous motor (PMSM) control using ADRC (learn more in this video)
  • Demo 2: House heating system control using data-driven MPC (learn more in this video)
  • Demo 3: Rotary inverted pendulum control using RL
Representations of demos presented in the Data-Driven Control webinar
Figure: Demos presented at Data-Driven Controls webinar.]]>
https://blogs.mathworks.com/deep-learning/2024/02/21/data-driven-control-with-matlab-and-simulink/feed/ 0
Verification and Validation for AI: Learning process verification https://blogs.mathworks.com/deep-learning/2024/02/07/verification-and-validation-for-ai-learning-process-verification/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2024/02/07/verification-and-validation-for-ai-learning-process-verification/#respond Wed, 07 Feb 2024 12:51:57 +0000 https://blogs.mathworks.com/deep-learning/?p=13907

The following post is from Lucas García, Product Manager for Deep Learning Toolbox.  This is the third post in a 4-post series on Verification and Validation (V&V) for AI. The series... read more >>

]]>
The following post is from Lucas García, Product Manager for Deep Learning Toolbox. 
This is the third post in a 4-post series on Verification and Validation (V&V) for AI.
The series began with an overview of V&V’s importance and the W-shaped development process, followed by a practical walkthrough in the second post, detailing the journey from defining AI requirements to training a robust pneumonia detection model.
This post is dedicated to learning process verification. We will show you how to ensure that specific verification techniques are in place to guarantee that the pneumonia detection model trained in the previous blog post meets the identified model requirements.
Steps in W-shaped development process with highlighted the step of Learning Process Verification
Figure 1: W-shaped development process, highlighting the stage covered in this post. Credit: EASA, Daedalean.
 

Testing and Understanding Model Performance

The model was trained using fast gradient sign method (FGSM) adversarial training, which is a method for training networks so that they are robust to adversarial examples. After training the model, particularly following adversarial training, it is crucial to assess its accuracy using an independent test set.
The model we developed achieved an accuracy exceeding 90%, which not only meets our predefined requirement but also surpasses the benchmarks reported in the foundational research for comparable neural networks. To gain a more nuanced understanding of the model’s performance, we examine the confusion matrix, which sheds light on the types of errors the model makes.
Confusion chart for adversarially-trained model showing accuracy of 90.71%, and true and predicted classes
Figure 2: Confusion chart for the adversarially-trained model.
Explainability techniques like Grad-CAM offer a visual understanding of the influential regions in the input image that drive the model’s predictions, enhancing interpretability and trust in the AI model’s decision-making process. Grad-CAM highlights the regions of the input image that contributed most to the final prediction.
Two images of lungs with pneumonia. The left image is showing the ground truth and the right image is showing the prediction with Grad-CAM.
Figure 3: Understanding network predictions using Gradient-weighted Class Activation Mapping (Grad-CAM).
 

Verify Robustness of Deep Learning Models

Adversarial Examples

Robustness of the AI model is one of the main concerns when deploying neural networks in safety-critical situations. It has been shown that neural networks can misclassify inputs due to small imperceptible changes.
Consider the case of an X-ray image that a model correctly identifies as indicative of pneumonia. When a subtle perturbation is applied to this image (that is, a small change is applied to each pixel of the image), the model’s output shifts, erroneously classifying the X-ray as normal.
Effect of input perturbation to lung image with pneumonia. The classifier misclassifies the image as normal.
Figure 4: Adversarial examples: effect of input perturbation to image classification.
 

L-infinity norm

To understand and quantify these perturbations, we turn to the concept of the l-infinity norm.
Imagine you have a chest X-ray image. A perturbation with an l-infinity norm of, say, 5 means adding or subtracting any number from 0 to 5 to any number of pixels. In one scenario, you might add 5 to every pixel within a specific image region. Alternatively, you could adjust various pixels by different values within the range of -5 to 5 or alter just a single pixel.
Examples of input perturbations of a pixel of a lung image.
Figure 5: L-infinity norm: examples of possible input perturbations.
However, the challenge is that we need to account for all possible combinations of perturbations within the -5 to 5 range, which essentially presents us with an infinite number of scenarios to test. To navigate this complexity, we employ formal verification methods, which provide a systematic approach to testing and ensuring the robustness of our neural network against a vast landscape of potential adversarial examples.

Formal verification

Given one of the images in the test set, we can choose a perturbation that defines a collection of perturbed images for this specific image. It is important to note that this collection of images is extremely large (the images depicted in the volume in Figure 5 are just a representative sample), and it is not practical to test each perturbed image individually.
Deep Learning Toolbox Verification Library allows you to verify and test robustness of deep learning networks using formal verification methods, such as abstract interpretation. The library enables you to verify whether the network you have trained is adversarially robust with respect to the class label provided an input perturbation.
Abstract interpretation applied to a lung image. The classification results can be interpreted as verified, unproven, or violated.
Figure 6: Formal verification using abstract interpretation.
Formal verification methods offer a mathematical approach that may be used to have formal proof of the correctness of a system. It allows us to conduct rigorous tests across the entire volume of perturbed images to see if the network’s output is affected. There are three potential outcomes for each of the images:
  • Verified - The output label remains consistent.
  • Violated - The output label changes.
  • Unproven - Further verification efforts or model improvement is needed.
Let’s set up the perturbation for our specific problem. The image values in our test set (XTest) range from 0 to 1. We set the perturbation to 1%, up or down. We set the perturbation bounds by using XLower and XUpper and define a collection of images (i.e., the volume in Figure 5). This means that we will test all possible perturbations of images that fall within these bounds.
Before running the verification test, we must convert the data to a dlarray object. The data format for the dlarray object must have the dimensions “SSCB” (spatial, spatial, channel, batch) to represent 2-D image inputs. Note that XTest is not just a single image but a batch of images to verify. So, we have a volume to verify for each of the images in the test set.
perturbation = 0.01; 
XLower = XTest - perturbation; 
XUpper = XTest + perturbation; 
XLower = dlarray(XLower,"SSCB"); 
XUpper = dlarray(XUpper,"SSCB"); 
We are now ready to use the verifyNetworkRobustness function. We specify the trained network, the lower and upper bounds, and the ground truth labels for the images.
result = verifyNetworkRobustness(net,XLower,XUpper,TTest); 
summary(result) 

verified 402

violated 13

unproven 209

The outcome reveals over 400 images verified, 13 violations, and more than 200 unproven results. We’ll have to go back to those images where the robustness test returned violated or unproven results and see if there is anything we can learn. But for over 400 images, we were able to formally prove that no adversarial example within a 1% perturbation range alters the network’s output—and that’s a significant assurance of robustness.
Another question that we can answer with formal verification is if adversarial training contributed to network robustness. In the second post of the series, we began with a reference model and investigated various training techniques, ultimately adopting an adversarially trained model. Had we used the original network, we would have faced unproven results for nearly all images. And in a safety-critical context, you’ll likely need to treat the unproven results as violations. While data augmentation contributed to verification success, adversarial training enabled the verification of substantially more images, leading to a superiorly robust network that satisfies our robustness requirements.
Bar graph showing number of observations for each verification result (verified, violated, and unproven) for original network, data-augmented network, and robust network.
Figure 7: Comparing verification results from various trained networks.
 

Out-of-Distribution Detection

A trustworthy AI system should produce accurate predictions in a known context. Still, it should also be able to identify unknown examples to the model and reject them or defer them to a human expert for safe handling. Deep Learning Toolbox Verification Library also includes functionality for out-of-distribution (OOD) detection.
Consider a sample image from our test set. To evaluate the model’s ability to handle OOD data, we can derive new test sets by applying meaningful transformations to the original images, as shown in the following figure.
Deriving datasets by adding speckle noise, FlipLR transformation, and contrast transformation to a lung image.
Figure 8: Derived datasets to explore out-of-distribution detection.
Using this library, you can create an out-of-distribution data discriminator to assign confidence to network predictions by computing a distribution confidence score for each observation. It also provides a threshold for separating the in-distribution from the out-of-distribution data.
In the following chart, we observe the network distribution scores for the training data, represented in blue, which constitutes the in-distribution dataset. We also see scores for the various transformations applied to the test set.
Bar graph of relative percentage versus distribution confidence scores for training data, speckle noise, FlipLR, and contrast.
Figure 9: Distribution of confidence scores for the original and derived datasets.
By using the distribution discriminator and the obtained threshold, when the model has to classify images with a transformation at test time, we can tell that if the images would be considered in- or out-of- distribution. For example, the images with speckle noise (see Figure 8), would be in-distribution, so we could trust the network output. On the contrary, the distribution discriminator considers the images with the FlipLR and contrast transformations (also see Figure 8) as out-of-distribution, so we shouldn’t trust the network output in those situations.
 

What’s Next?

Stay tuned for our fourth and final blog post, where we will navigate the right-hand side of the W-diagram, focusing on deploying and integrating our robust pneumonia detection model into its operational environment. We will show how to bridge the gap between a well-trained model and a fully functional AI system that can be trusted in a clinical setting.]]>
https://blogs.mathworks.com/deep-learning/2024/02/07/verification-and-validation-for-ai-learning-process-verification/feed/ 0
Large Language Models with MATLAB https://blogs.mathworks.com/deep-learning/2024/01/22/large-language-models-with-matlab/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2024/01/22/large-language-models-with-matlab/#respond Mon, 22 Jan 2024 14:51:45 +0000 https://blogs.mathworks.com/deep-learning/?p=13676

How to connect MATLAB to the OpenAI™ API to boost your NLP tasks. Have you heard of ChatGPT™, Generative AI, and large-language models (LLMs)? This is a rhetorical question at this point. But did... read more >>

]]>

How to connect MATLAB to the OpenAI™ API to boost your NLP tasks.

Have you heard of ChatGPT™, Generative AI, and large-language models (LLMs)? This is a rhetorical question at this point. But did you know you can combine these transformative technologies with MATLAB? In addition to the MATLAB AI Chat Playground (learn more by reading this blog post), you can now connect MATLAB to the OpenAI™ Chat Completions API (which powers ChatGPT).
In this blog post, we are talking about the technology behind LLMs and how to connect MATLAB to the OpenAI API. We also show you how to perform natural language processing (NLP) tasks, such as sentiment analysis and building a chatbot, by taking advantage of LLMs and tools from Text Analytics Toolbox.
 

What are LLMs?

Large language models (LLMs) are based on transformer models (a special case of deep learning models). Transformers are designed to track relationships in sequential data. They rely on a self-attention mechanism to capture global dependencies between input and output. LLMs have revolutionized NLP, because they can capture complex relationships between words and nuances present in human language.
Well known transformer models include BERT and GPT models, both of which you can use with MATLAB. If you want to use a pretrained BERT model included with MATLAB, you can use the bert function. In this blog post, we are focusing on GPT models.
 

LLMs Repository

The code you need to access and interact with LLMs using MATLAB is in the LLMs repository. By using the code in the repository, you can interface the ChatGPT API from your MATLAB environment. Some of the supported models are gpt-3.5-turbo and gpt-4.
GitHub repository for large language models with MATLAB
 

Set Up

To interface the ChatGPT API, you must obtain an OpenAI API key. To learn more about how to obtain the API key and charges for using the OpenAI API, see OpenAI API. It’s good practice to save the API key in a file in your current folder, so that you have it handy.
How to save OpenAI API key for reuse
Animated Figure: Save the OpenAI API key in your current folder.
 

Getting Started

To initialize the OpenAI Chat object and get started with using LLMs with MATLAB, type just one line of code.
chat = openAIChat(systemPrompt,ApiKey=my_key);
In the following sections of this blog, I will show you how to specify the system prompt for different use cases and how to enhance the functionality of the OpenAI Chat object with optional name-value arguments.

Getting Started in MATLAB Online

You might want to work with LLMs in MATLAB Online. GitHub repositories with MATLAB code have an “Open in MATLAB Online” button. By clicking on the button, the repository opens directly in MATLAB Online. Watch the following video to see how to open and get started with the LLMs repository in MATLAB Online in less than 30 seconds.
Open large language models repository in MATLAB Online
Animated Figure: Use LLMs in MATLAB Online.
 

Use Cases

In this section, I am going to present use cases for LLMs with MATLAB and link to relevant examples. The use cases include sentiment analysis, building a chatbot, and retrieval augmented generation. You can use tools from Text Analytics Toolbox to preprocess, analyze, and ‘meaningfully’ display text. We are going to mention a few of these functions here but check the linked examples to learn more.

Sentiment Analysis

Let’s start with a simple example on how to perform sentiment analysis. Sentiment analysis deals with the classification of opinions or emotions in text. The emotional tone of the text can be classified as positive, negative, or neutral.
Sentiment analysis with large language models
Figure: Creating a sentiment analysis classifier.
 
Specify the system prompt. The system prompt tells the assistant how to behave, in this case, as a sentiment analyzer. It also provides the system with simple examples on how to perform sentiment analysis.
systemPrompt = "You are a sentiment analyser. You will look at a sentence and output"+...
    " a single word that classifies that sentence as either 'positive' or 'negative'."+....
    "Examples: \n"+...
    "The project was a complete failure. \n"+...
    "negative \n\n"+...  
    "The team successfully completed the project ahead of schedule."+...
    "positive \n\n"+...
    "His attitude was terribly discouraging to the team. \n"+...
    "negative \n\n";
Initialize the OpenAI Chat object by passing a system prompt.
chat = openAIChat(systemPrompt,ApiKey=my_key);
Generate a response by passing a new sentence for classification.
text = generate(chat,"The team is feeling very motivated.")

  text = "positive"

The text is correctly classified as having a positive sentiment.

Build Chatbot

A chatbot is software that simulates human conversation. In simple words, the user types a query and the chatbot generates a response in a natural human language.
Building a chatbot with large language models
Figure: Building a chatbot.
 
Chatbots started as template based. Have you tried querying a template-based chatbot? Well, I have and almost every chat ended with me frantically typing “talk to human”. By following the Example: Build ChatBot in the LLMs repository, I was able to build a helpful chatbot in minutes.
The first two steps in building a chatbot are to (1) create an instance of openAIChat to perform the chat and (2) use the openAIMessages function to store the conversation history.
chat = openAIChat("You are a helpful assistant. You reply in a very concise way, keeping answers "+...
    "limited to short sentences.",ModelName=modelName,ApiKey=my_key);
messages = openAIMessages;
After a few more lines of code, I built a chatbot that helped me plan my Mexico vacations. In addition to the example code, I used other MATLAB functions (e.g., extractBetween) to format the chatbot responses. The following figure shows my brief (but helpful) chat with the chatbot. Notice that the chatbot retains information from previous queries. I don’t have to repeat “Yucatan Peninsula” in my questions.
Conversation with chatbot built with large language model
Figure: User queries and chatbot responses for planning a Mexico vacation.
 

Retrieval Augmented Generation

Retrieval-augmented generation (RAG) is a technique for enhancing the results achieved by an LLM. Both accuracy and reliability can be augmented by retrieving information from external sources. For example, the prompt fed to the LLM can be enhanced with more up-to-date or technical information.
Workflow for retrieval augmented generation (RAG) with large language model
Figure: Workflow for retrieval-augmented generation (RAG).
 
The Example: Retrieval-Augmented Generation shows how to retrieve information from technical reports on power systems to enhance ChatGPT for technical queries. I am not going showing all the example details here, but I will highlight key steps.
  1. Use MATLAB tools (e.g., websave and fileDatastore) for retrieving and managing online documents.
  2. Use Text Analytics Toolbox functions (e.g., splitParagraphs, tokenizedDocument, and bm25Similarity) for preparing the text from the retrieved documents.
  3. When the retrieved text is ready for the task, initialize the chatbot with the specified context and API key.
    chat = openAIChat("You are a helpful assistant. You will get a " + ...
        "context for each question, but only use the information " + ...
        "in the context if that makes sense to answer the question. " + ...
        "Let's think step-by-step, explaining how you reached the answer.",ApiKey=my_key);
    
  4. Define the query. Then, retrieve and filter the relevant documents based on the query.
    query = "What technical criteria can be used to streamline new approvals for grid-friendly DPV?";
    selectedDocs = retrieveAndFilterRelevantDocs(allDocs,query);
    
  5. Define the prompt for the chatbot and generate a response.
    prompt = "Context:" ...
        + join(selectedDocs, " ") + newline +"Answer the following question: "+ query;
    response = generate(chat,prompt);
    
    Wrap the text for easier visualization.
    wrapText(response)
    

    ans =

    "The technical criteria that can be used to streamline new approvals for grid-friendly DPV can include prudent screening criteria for systems that meet certain specifications. These criteria can be based on factors such as DPV capacity penetration relative to minimum feeder daytime load. Additionally, hosting capacity calculations can be used to estimate the point where DPV would induce technical impacts on system operations. These screening criteria are commonly used in countries like India and the United States."

Other Use Cases

The use cases presented above are just a sample of what you can achieve with LLMs. Other notable use cases (with examples in the LLMs repository) include text summarization and function calling. You can also use LLMs for many other NLP tasks like machine translation.
What will you use the MATLAB LLMs repository for? Leave comments below and links to your GitHub repository.
Text summarization is automatically creating a short, accurate, and legible summary of a longer text document. In the Example: Text Summarization, you can see how to incrementally summarize a large text by breaking it into smaller chunks and summarizing each chunk step by step.
Function calling is a powerful tool that allows you to combine the NLP capabilities of LLMs with any functions that you define. But remember that ChatGPT can hallucinate function names, so avoid executing any arbitrary generated functions and only allow the execution of functions that you have defined. For an example on how to use function calling for automatically analyzing scientific papers from the arXiv API, see Function Calling with LLMs.
 

Key Takeaways

  1. There is a new GitHub repository that enables you to use GPT models with MATLAB for natural language processing tasks. Find the repository here.
  2. The repository includes example code and use cases that achieve many tasks like sentiment analysis and building a chatbot.
  3. Take advantage of MATLAB tools, and more specifically Text Analytics Toolbox functions, to enhance the LLM functionality, such as retrieving, managing, and preparing text.
]]>
https://blogs.mathworks.com/deep-learning/2024/01/22/large-language-models-with-matlab/feed/ 0
Lidar Code-Along https://blogs.mathworks.com/deep-learning/2024/01/16/lidar-code-along/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2024/01/16/lidar-code-along/#respond Tue, 16 Jan 2024 14:20:21 +0000 https://blogs.mathworks.com/deep-learning/?p=13616 Lidar (light detection and ranging) is a remote sensing technology. Lidar sensors emit laser pulses that reflect off objects, allowing them to perceive the structure of their surroundings. The... read more >>

]]>
Lidar (light detection and ranging) is a remote sensing technology. Lidar sensors emit laser pulses that reflect off objects, allowing them to perceive the structure of their surroundings. The sensors record the reflected light energy to determine the distances to objects to create a 2D or 3D representations of the surroundings.
With lidar technology a point cloud is created, that is a collection of data points plotted in 3-D space, where each point represents the X-, Y-, and Z-coordinates of a location on a real-world object’s surface, and the points collectively map the entire surface. Lidar is used in many industries including forestry, infrastructure analysis, and mining.
Deep learning adds precision and speed to the processing of point cloud data. Watch the following five videos that walk you through the steps of a lidar processing workflow with deep learning. You can also find the videos here.
For more examples, see lidar with deep learning examples. To learn more about how MATLAB users can use deep learning, computer vision, and image processing for addressing lidar challenges, check out this story: Spacesium Creates Deep Learning System to Segment Large Lidar Point Clouds with MATLAB.
 

Load Point Cloud Data

Load point cloud data as a data store using the pcread function and load the bounding box labels using the boxLabelDatastore function.
 

Preprocess Data

Split the data into training and testing sets. Synthetically increase the size of the training data set by randomly adding a fixed number of car and truck class objects to every point cloud and by using augmentation techniques such as flipping, scaling, rotation, and translation. To learn more about typical data augmentation techniques for 3-D object detection workflows with lidar data, see Data Augmentations for Lidar Object Detection Using Deep Learning.
 

Define Deep Neural Network

Define network parameters, such as anchor boxes and pillars, for the PointPillars network. Then, define the PointPillars detection network using the pointPillarObjectDetector function.
 

Train Network

To train a network, you must specify training options. Because training a network can be time consuming, a pretrained PointPillars model is used for this workflow.
 

Object Detection

Test the PointPillars network on a test dataset and display the detected output point cloud with bounding boxes.
 ]]>
https://blogs.mathworks.com/deep-learning/2024/01/16/lidar-code-along/feed/ 0
NeurIPS Highlights https://blogs.mathworks.com/deep-learning/2023/12/21/neurips-highlights/?s_tid=feedtopost https://blogs.mathworks.com/deep-learning/2023/12/21/neurips-highlights/#respond Thu, 21 Dec 2023 14:36:07 +0000 https://blogs.mathworks.com/deep-learning/?p=13495

We are back from the NeurIPS 2023 conference. It was truly inspiring to engage with the AI community and learn more about the current developments and future of AI. We also enjoyed the opportunity... read more >>

]]>
We are back from the NeurIPS 2023 conference. It was truly inspiring to engage with the AI community and learn more about the current developments and future of AI. We also enjoyed the opportunity to share how you can use MATLAB to integrate AI into cyber-physical systems. In this blog post, I am going to present some highlights from the MATLAB AI team’s presence at the conference.
MATLAB AI team at MathWorks booth at NeurIPS 2023
Figure: Our team at the MathWorks booth
Left to right (front): Ashvant Ram SelvamLucas GarcíaBrenda ZhuangSivylla ParaskevopoulouAnoush NajarianMaitreyi Chitale, and Jon Cherrie
Left to right (back): Mary Ann Freeman, Antoni Woss, Naren Srivaths Raman, Conor Daly, Jianghao Wang, Kevin Holly, Elvira Osuna Highley, and Abhijit Bhattacharjee
 

3 Talks and A Poster

Here, you can find an overview of the talks and poster presented by MathWorks at NeurIPS, including links to slides.
MathWorks talks and poster at NeurIPS with large number of attendees
Figure: Lucas García and Naren Srivaths Raman presenting talks on Incorporating ML Models into Safety-Critical Systems and Reinforcement Learning: Trends, Applications, and Challenges, and Maitreyi Chitale presenting a poster on Farm-To-Plate-AI.
 

Talk 1

From Theory to Practice: Incorporating ML Models into Safety-Critical Systems
Lucas García
In this talk, Lucas delved into the complexities of certifying neural networks for safety-critical applications, with a particular focus on the aviation industry. The presentation introduced AI Certification in the aviation industry, and a case study for a runway sign classification system in the cockpit designed to improve pilot navigation and situational awareness. By walking the audience through each step of the workflow—from requirement definition to model training and system validation—the talk provided a comprehensive blueprint for integrating neural networks into safety-critical systems for low criticality applications, paving the way for future advancements in industry-specific certification standards.
Slides
 

Talk 2

Using AI to Improve Control Design Workflows
Naren Srivaths Raman
Control systems are ubiquitous and enable the safe and predictable operation of airplanes, cars, and energy systems. Just as in other engineering disciplines, control engineers are interested in the new possibilities AI offers to enhance traditional solutions. The talk covered several areas where AI is gaining interest and adoption among control engineers and researchers. In this talk, Naren (1) explored AI for modeling a system controlled (plant) with techniques such as nonlinear system identification and reduced order modeling (ROM), (2) discussed the use of AI for virtual sensor modeling and control algorithm design, in particular, the design of nonlinear model predictive control (MPC) using neural state-space (NSS) models, and (3) touched upon how reinforcement learning (RL) can be used as a tool for controller tuning or can replace a traditional controller altogether. Finally, Naren introduced some best practices in using AI for controls to address challenges, such as lack of performance guarantees.
Slides
 

Talk 3

Reinforcement Learning: Trends, Applications, and Challenges
Naren Srivaths Raman
Reinforcement learning (RL) has been gaining attention as a machine learning technique that can automatically learn complex behaviors and realize high performance. RL applications span various domains, including control design, robotics, automated driving, communications, and more. However, reinforcement learning comes with several challenges. These include the need for large amounts of training data, difficulties in tuning hyperparameters, and verification of deep neural network policies. In this talk, Naren discussed trends, applications, and challenges, and introduced ideas, tools, and best practices on how to address these challenges.
Slides
 

Poster

Farm-to-Plate-AI: Enhancing Freshness and Reducing Waste using Computer Vision and Robotics
Maitreyi Chitale, Brenda Zhuang, et al.
Addressing global food security and reducing waste in the food supply chain are crucial challenges in today's world. The poster highlights the transformative potential of integrating emerging technologies, such as autonomous navigation, artificial intelligence (AI), and computer vision, to optimize farming practices, automate harvesting and grading, and monitor food quality during transportation. By leveraging these technologies, we can effectively address the challenges posed by conflicts, pandemics, climate change, inequality, and the projected 70% increase in food demand due to population growth. We built a workshop with exercises to simulate the pipeline, which included simulating a UAV through different farm configurations, utilizing YOLOv4 and performing transfer learning on it with a smaller dataset to perform object counting of mangoes pre-harvest, and utilizing machine learning on hyperspectral images to gauge the ripeness of mangoes. The workshop was presented at GHC23 to 750+ in-person participants, as well as at virtual WiDS workshop series and beyond. Our poster, which is based on the workshop we created, proposes an end-to-end pipeline, from farm to plate and leverages the combined gains of individual AI and robotics algorithms to optimize the global food supply chain and achieve significant waste reduction.
 

Demos

As promised (in a previous blog post) , we had lots of goodies at our both and cool demos showing the AI Chat Playground (the new GenAI tool for generating MATLAB code), real-time instance segmentation on a GPU, and the crowd favorite demo - a robot playing Tic-Tac-Toe. An RL agent was trained to play the game and the robot used computer vision for recognizing the progress of the Tic-Tac-Toe game on the board.
Robotic arm using reinforcement learning and computer vision to play Tic-Tac-Toe
Figure: (left) The robot arm and Naren, the robot programmer. (right) Elvira playing Tic-Tac-Toe against the robot.
Watch the video below to see the robot in action.
Video (4x): Robot using reinforcement learning and computer vision is playing Tic-Tac-Toe.
 

See You Next Year

We are looking forward to joining NeurIPS again next year for more sharing, events, and booth interactions. Until then, leave your comment here to talk about AI, your workflows, and challenges.
MathWorks team at WiML event
Figure: At the WiML (Women in Machine Learning) event at NeurIPS.
]]>
https://blogs.mathworks.com/deep-learning/2023/12/21/neurips-highlights/feed/ 0
Podcast Alert: Deploying Edge and Embedded AI Systems https://blogs.mathworks.com/deep-learning/2023/12/12/podcast-alert-deploying-edge-and-embedded-ai-systems/?s_tid=feedtopost Wed, 13 Dec 2023 04:01:31 +0000 https://blogs.mathworks.com/deep-learning/?p=13465

The following blog post is from Daniel Prieve, Digital Marketing Manager. Last month, Heather Gorr was interviewed for the TWIML AI Podcast (hosted by Sam Charrington). Heather shared knowledge,... read more >>

]]>
The following blog post is from Daniel Prieve, Digital Marketing Manager.
Last month, Heather Gorr was interviewed for the TWIML AI Podcast (hosted by Sam Charrington). Heather shared knowledge, which she has gained as a MATLAB Product Manager, on how to prepare and test AI models before deploying the models to edge devices and embedded systems.
You can find the podcast on “Deploying Edge and Embedded AI Systems”, here:
Podcast on deploying edge and embedded AI systems
In this blog post, we highlight a few key points from the TWIML podcast on Edge AI. But you will certainly learn a lot more by listening to the full podcast.
Data Preparation: When you prepare data for training and testing an AI model that will later be deployed to the edge, you must take into consideration hardware limitations and how they will impact the quality and streaming process of the data. This is particularly important when streaming data captured by sensors.
Model Preparation: Research models might perform well on the desktop, but AI practitioners should consider additional steps before deploying their models to the edge. They need to consider (1) compressing the models (for example, with quantization) to ensure the models will fit on the target device, (2) applying explainability techniques to add transparency to AI decisions, and (3) verify the models’ robustness with testing and validation (for example, verifying robustness against adversarial examples).
Simulation: Simulating an AI model in a system-wide context before deployment tests how well the model integrates with other parts of the system. By simulating a physical system, you can generate synthetic data when enough data is not available for training an AI model.
Collaborative Effort: Embedding AI models into hardware systems requires collaboration across many teams: hardware experts, data scientists, and domain-focused engineers. These teams, which might be working on different platforms and have different skill sets, must be given the right tools for successful communication, collaboration, and sharing outcomes.
Is deploying AI models to the edge part of your workflow? And what do you spend most of your time on: data preparation, model preparation, or system simulation? What have you learned in the podcast that you can apply back into your work? Share your comments and thoughts below.]]>
MATLAB Is Coming to NeurIPS https://blogs.mathworks.com/deep-learning/2023/12/05/matlab-is-coming-to-neurips/?s_tid=feedtopost Tue, 05 Dec 2023 15:15:12 +0000 https://blogs.mathworks.com/deep-learning/?p=13438

NeurIPS 2023 is starting soon; this Sunday December 10th. I am very excited to join the conference along with other members of the MATLAB AI team and reach three personal firsts (not necessarily in... read more >>

]]>
NeurIPS 2023 is starting soon; this Sunday December 10th. I am very excited to join the conference along with other members of the MATLAB AI team and reach three personal firsts (not necessarily in order of importance):
  1. Visit New Orleans and have authentic beignets.
  2. Attend the biggest machine learning conference.
  3. Be at a conference as a non-academic.
Come see me and the rest of the AI team at the MathWorks booth to talk about AI and how to integrate AI in engineered systems, check out our demos, and get free goodies.
And of course, you shouldn’t miss the talks by my colleagues on incorporating machine learning models into safety critical-systems, reinforcement learning, and using AI for control design. All three talks will be presented on Sunday the 10th.
To get a sneak peek at the topic of safety-critical systems, check out our previous blog posts: The Road to AI Certification: The importance of Verification and Validation in AI and Verification and Validation for AI: From requirements to robust modeling.
Hope to see you at NeurIPS and if you are not joining, you can always leave a comment below to talk about AI and how you integrate it in your workflows.]]>
Visual Inspection Code-Along https://blogs.mathworks.com/deep-learning/2023/11/28/visual-inspection-code-along/?s_tid=feedtopost Tue, 28 Nov 2023 15:38:27 +0000 https://blogs.mathworks.com/deep-learning/?p=13401 Visual inspection is the image-based inspection of parts where a camera scans the part under test for failures and quality defects. By using deep learning and computer vision techniques, visual... read more >>

]]>
Visual inspection is the image-based inspection of parts where a camera scans the part under test for failures and quality defects. By using deep learning and computer vision techniques, visual inspection can be automated for detecting manufacturing flaws in many industries such as biotech, automotive, and semiconductors. For example, with visual inspection, flaws can be detected in semiconductor wafers and pills.
Watch the following video that walks you through the steps of a visual inspection workflow using a ResNet convolutional neural network. MATLAB provides access to many pretrained deep learning models that you can use for visual inspection. For more visual inspection examples, see Automated Visual Inspection.
]]>
PyTorch Models and Git in MATLAB Online https://blogs.mathworks.com/deep-learning/2023/11/07/pytorch-models-and-git-in-matlab-online/?s_tid=feedtopost Tue, 07 Nov 2023 16:40:05 +0000 https://blogs.mathworks.com/deep-learning/?p=13337

The following post is from Yann Debray, Product Manager for MATLAB Online, and Sivylla Paraskevopoulou, Product Marketing Manager for AI. Do you use MATLAB Online to create AI workflows? Do you... read more >>

]]>
The following post is from Yann Debray, Product Manager for MATLAB Online, and Sivylla Paraskevopoulou, Product Marketing Manager for AI.
Do you use MATLAB Online to create AI workflows? Do you want to write trackable and shareable code?
This blog post shows you how to (1) import models from PyTorch® using MATLAB Online, (2) use Git™ source control in MATLAB Online, and (3) open MATLAB Online directly from GitHub®. For each of the steps of the workflow, we have included a short animation to help you replicate the step.
Import PyTorch models in MATLAB Online and integrate with GitHub
 

Create and Clone GitHub Repository

We first created a new GitHub repository with a README file. We cloned this repository in MATLAB Online, which copied all the files from the repository to a folder in our working directory.
Animated Figure: Create a GitHub repository and clone it with MATLAB Online.
We generated a personal access token to use in place of a password. We are going to need the token to authenticate with GitHub when pushing our new or updated files to the GitHub repository.
Animated Figure: Generate personal access token.
 

Import Model from PyTorch in MATLAB Online

Starting in MATLAB R2023b, you can import models from PyTorch in MATLAB Online. To show this functionality, we reused the code from the doc example Import Network from PyTorch and Classify Image. Once we confirmed that the code runs as expected (that is, the PyTorch model is successfully imported and the test image is correctly classified), in three quick steps we uploaded the code to our GitHub repository.
  1. We added the live script to Source Control.
  2. We committed the script.
  3. We pushed the script to the GitHub repository.
Observe how fast and intuitive using Git source control in MATLAB Online is!
Animated Figure: Push live script (importing PyTorch model) from MATLAB Online to your GitHub repository.
Then, we did a quick update to the repository’s README from MATLAB Online and pushed the changes to the repository.
Animated Figure: Push updated README.
 

Add MATLAB Online Button to Repository

Finally, we added an “Open in MATLAB Online” button to our GitHub repository. By clicking on the button, the GitHub repository opens directly in MATLAB Online (independently from the Git-tracked folder that we previously created). This makes it easy to share your code with your team or community. To learn more, see Open in MATLAB Online from GitHub.
Animated Figure: Open code from GitHub repository with MATLAB Online.
]]>