Giving LLMs new capabilities: Ollama tool calling in MATLAB
At their heart, LLMs are just text generation machines that predict the next word based on patterns learned from huge amounts of data. If you want them to do something more than this such as perform computation reliably, look up today's weather or quote up to date stock prices, you need to give them access to tools. Tools are simply functions that give Large Language Models (LLMs) extra capabilities.
Today, I'm going to show how to call tools defined in MATLAB from a local Large Language Model called mistral-nemo running on my desktop via Ollama. To follow along, you will need to
- Install the Large Language Models (LLMs) with MATLAB support package
- Install Ollama and the mistral-nemo model. My article How to run local DeepSeek models and use them with MATLAB might help here, just replace deepseek-r1:1.5b with mistral-nemo in the instructions
Once you've done this, everything will run on your local machine.
Small models show the limitations of LLMs much more readily
As a math geek, I have been torturing LLMs with maths problems from the very beginning and it always amused me when the supposedly superhuman AIs got even the most basic arithmetic problems wrong. It seems I am not alone! A simple search of 'arithmetic chattgpt' on BlueSky furnished me with many amusing examples showing how it can fail to add up a list of numbers, count the number of 'r's in 'blueberry' and so on.
The fact that LLMs make mistakes like this is not surprising at all once you understand how they work. LLMs don't perform calculations, they predict patterns based on their training data. "1+1=2" will have appeared in the training data a lot. "590675249 + 9579089", not so much. If anything, I am surprised that LLMs work as well as they do on this sort of thing and they are getting better all the time! As the saying goes 'Today's AI is the worst you'll ever use...'.
mistral-nemo is tiny compared to the models you've likely used online such as ChatGPT or Claude. It has 'only' 12 billion parameters which means that I can run it on my aging desktop! It's surprisingly capable but it's also pretty easy to demonstrate how it can fail at arithmetic.
chat = ollamaChat("mistral-nemo");
generate(chat,"Find the sum of 590675249 and 9579089")
Is it sure? Let's ask it again!
generate(chat,"Find the sum of 590675249 and 9579089")
Now lets simply use MATLAB
590675249 + 9579089
The model got it right the second time and wrong the first time although your results may vary. This neatly demonstrates a couple facts about LLMs which are:-
- They are stochastic.
- They can get things wrong and do so very confidently. These mistakes are often called hallucinations and they seem to be a fundamental limitation of LLM based technology. Larger models can reduce how often they happen but hallucination is inevitable.
Given issues like this, it can be tempting to dismiss LLM based artificial intelligence completely. It can get even very basic things wrong while presenting results confidently as fact. Well, so can I and MathWorks seems to think that, on balance, I'm useful enough to pay a salary. We need to take the good parts of these technologies and engineer our way around the limitations.
I like to think that I walk a reasonable line when it comes to LLM-based AI technology. I am excited by the possibilities, use it all the time but keep in mind its weaknesses and limitations at all times. You know...like any other tool we've ever used for anything ever.
Defining a tool to give the LLM a new capability
Let's teach our LLM how to run a MATLAB function to help it out when it's asked to do something that it otherwise couldn't do well. This could be anything you might imagine but lets start simple:
function result = addTwoNumbers(a,b)
result = a+b;
end
To make it available to our Ollama based LLM, we (confusingly) use the openAIFunction from the Large Language Models (LLMs) with MATLAB support package. The choice of function name is from back when only OpenAI was offering this functionality.
toolAddTwoNumbers = openAIFunction("addTwoNumbers", "Add two numbers using MATLAB");
toolAddTwoNumbers = addParameter(toolAddTwoNumbers,"x1",type="number");
toolAddTwoNumbers = addParameter(toolAddTwoNumbers,"x2",type="number");
Now, when we create a chat object, we supply it with the tool we've just defined
agent = ollamaChat("mistral-nemo",tools=toolAddTwoNumbers);
Let's ask our new, augmented LLM to do some addition
[~,response] = generate(agent,"Find the sum of 590675249 and 9579089")
This underwhelming looking result occurs because the LLM can't directly run any code! What it has done instead is realize that it needs to call the tool we've defined and returned exactly how it wants to call it by adding a tool_calls field to its response. It looks like this
response.tool_calls.function
response.tool_calls.function.arguments
So, when writing our simple AI agent, we test for the existence of a tool_calls field in the response and if we find one, we execute the tool as the AI had requested. If there isn't any call to a tool, we just show the AI response.
if isfield(response,"tool_calls")
fprintf("The AI has asked to call MATLAB function %s\n",response.tool_calls.function.name);
fprintf("With arguments: %f and %f\n",response.tool_calls.function.arguments.x1,response.tool_calls.function.arguments.x2);
result = addTwoNumbers(response.tool_calls.function.arguments.x1 , response.tool_calls.function.arguments.x2);
fprintf("The result of the function call is %f\n",result)
else
fprintf("The AI chose not to make a tools_call and provided the following response\n")
fprintf("%s",response.content)
end
A more advanced workflow might pass this tool output back to the AI for further processing but I'm keeping things as simple as possible for now.
Putting it all together into a simple AI agent
I've put all of this together into a new function called aiAdder which is defined at the end of this post. We know it can add up two integers so let's see what else it can do.
aiAdder("Find the sum of fity-six and nine million, three hundred thousand and seventy tow")
I'm quite pleased with that! Without AI, creating a parser that could convert written english words into digits wouldn't have been much fun. It even handled the typo at the end. How about a little puzzle?
aiAdder("Add the boiling point of water in Fahrenheit to the number of piano keys on a full-sized piano")
The tool only gets called when I ask it to add things. Otherwise, it simply replies to me as normal
aiAdder("In one sentence, describe why an LLM is bad at arithmetic")
That's quite a lot of functionality for just a few lines of code! Here's the full aiAdder function
function aiAdder(Prompt)
function result = addTwoNumbers(a,b)
result = a+b;
end
toolAddTwoNumbers = openAIFunction("addTwoNumbers", ...
"Add two numbers using MATLAB");
toolAddTwoNumbers = addParameter(toolAddTwoNumbers,"x1",type="number");
toolAddTwoNumbers = addParameter(toolAddTwoNumbers,"x2",type="number");
agent = ollamaChat("mistral-nemo",tools=toolAddTwoNumbers);
[~,response] = generate(agent,Prompt);
if isfield(response,"tool_calls")
fprintf("The AI has asked to call MATLAB function %s\n",response.tool_calls.function.name);
fprintf("With arguments: %f and %f\n",response.tool_calls.function.arguments.x1,response.tool_calls.function.arguments.x2);
result = addTwoNumbers(response.tool_calls.function.arguments.x1 , response.tool_calls.function.arguments.x2);
fprintf("The result of the function call is %f\n",result)
else
fprintf("The AI chose not to make a tools_call and provided the following response\n")
fprintf("%s",response.content)
end
end
Next steps
This is about as basic as it gets for tool calling with LLMs but we all have to start somewhere. Some more advanced demonstrations of tool calling with LLMs in MATLAB can be found on the LLMs with MATLAB GitHub repository and can can expect to see more with time.
What applications do you have in mind for this technology?
- Category:
- artificial intelligence (AI),
- Gen-AI,
- Tutorials
Comments
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.