Hi there. I am super excited to introduce our new Developer Zone blog! I work in a group at MathWorks that is responsible for building the infrastructure and architecture that we require as a company to deliver software that is of high quality, on schedule, and meets customer requirements. In short, we develop the build, test, and integration tools needed to produce production grade software on multiple platforms and using many different technologies. My specific team is focused on test infrastructure and are the primary developers of MATLAB's unit test framework.
As you might imagine I am very interested in development approaches and what it takes to build production software. This blog is a recognition of the fact that we are not the only ones who have a passion for this. Indeed, we hear from many of you who are facing similar tasks building production grade software using MATLAB® as a critical piece of the technology stack, and we have literally thousands of world class MATLAB developers in house doing the same thing. I am excited to share some of the building blocks we use to develop robust, extendable, flexible software that is written in MATLAB, and I am likewise excited to hear how you are doing the same.
To start, for our blog's first post I thought it would be fitting to describe how MATLAB can interface with a CI system which is a must have for collaborating with others to build high quality software.
Wait, what am I talking about, CI system? Well, what do you think of when you hear the term "Continuous Integration"? Do you think of calculating anti-derivatives and definite integrals of continuous functions or interfacing with a system that helps build and test software? A quick google check shows that most people think the latter, although perhaps those involved in mathematics may skew differently.
Whatever the case may be, developing software using a continuous integration (CI) system has proven to be essential for production grade software. The services and safety net that these systems provide enable the creation of software that can rise to high levels of safety and quality.
So how is this done using MATLAB?
There are a few common purposes of continuous integration systems. Three important purposes are:
- Interfacing with a source control system
- Building software that requires a compilation step like C++ or Java®
- Testing software that has changed or been rebuilt
Connecting your CI system with your source control system depends on both systems and this can be done independently from MATLAB.
As for building software, there are many features of MATLAB which do indeed require a build step, such as writing MEX files, writing Java classes that work with MATLAB, or p-coding MATLAB files for distribution. However, probably the most common MATLAB activity is writing code using the MATLAB language, which does not require a build step. For now, let's scope this discussion to these types of MATLAB activities that do not have any artifacts to build (i.e. the MATLAB code itself is the artifact).
If you are producing and sharing MATLAB code files, and they are already configured in your source control system then great, we are almost there. We just need to make sure that when you change your MATLAB source code your tests get run in your CI System and trigger build failures. Well, that is what I want to explore in more depth today. Note that there are a variety of CI systems out there, but let's look at connecting MATLAB tests to Jenkins™, which is popular and integrates with a variety of systems.
To do this we need to:
- Configure Jenkins to launch MATLAB, run the tests, and close MATLAB
- Communicate the results of the test run from MATLAB to the Jenkins software
- Ride out our days relaxing on the beach instead of manually integrating our code changes
Launching MATLAB can be done by creating a new free-form project in Jenkins. This type of project will allow us to set up a simple build script to launch MATLAB and run the tests. In this case I've named the job "Run ALL THE TESTS".
Now we can create our build action to run a shell script.
The shell command that we use needs to simply start MATLAB and run something. We can use this simple shell command to just confirm we can get this MATLAB running. Use this script and run a test build in Jenkins:
/Applications/MATLAB_R2014b.app/bin/matlab -nodisplay -r "disp('Hello World!');exit"
Connect the Tests!
Great! We are connected. Next step is to actually run something in MATLAB. This is dependent upon on how you have your tests organized and precisely which tests you wish to run. This exercise demonstrates running tests that are located in a "testcases" package as well as its sub-packages. Here's the updated shell script:
/Applications/MATLAB_R2014b.app/bin/matlab -nodisplay -r "runALL_THE_TESTS"
and here is the content of the runAll_THE_TESTS script:
import matlab.unittest.TestSuite; try suite = TestSuite.fromPackage('testcases','IncludingSubpackages',true); results = run(suite); display(results); catch e disp(getReport(e,'extended')); exit(1); end exit;
So in short we want to create a TestSuite from our package and its sub-packages and then just run the tests using the default TestRunner. The try-catch usage is to allow MATLAB to exit quickly in the event of some unforeseen problem, but notice that before doing so the code prints the extended error details and exits with a non-zero status code. This will ensure that the Jenkins task will cause a build failure if something goes terribly wrong.
Let's run another job and see our tests in action
We are in the business of running tests! Unfortunately we are only running them and not reacting in any way to test failures. You can see from the blue job marker above that the Jenkins job actually passed despite many of the tests actually failing. This is no good! Looking at the bottom of the test log we see that 163 tests passed, 70 tests failed, and 80 tests did not complete, so why did the job pass?
What is happening here is that the Jenkins CI system does not know the results of this test run. The job determined pass or failure by looking at the exit status code of the MATLAB process we kicked off. While the tests failed, MATLAB was entirely happy as it started, ran the failing test suite and then exited cleanly. We need to tell MATLAB to exit with a non-zero status code if any test failures were encountered. We can do this by simply changing the last line of our run script to do just that:
Running the job we can see that we now see the test failures as job failures
For the win!
After just a few minutes we are now in business with MATLAB connected to our Jenkins CI system, running tests and failing jobs when those tests fail. Awesome! This will work in all releases that support the test framework (all the way back to R2013a), and now you can set up Jenkins run these builds on a regular basis, such as nightly or whenever a file is checked into the SCM system. If you have R2014a or later we can do even better, but I'll save that for the next blog post.
Do you automatically test your MATLAB code changes with a CI system? If so, which one? What benefits have you realized in your development process? What tips and tricks have you learned along the way? Let us know in the comments section below.
Get the MATLAB code
Published with MATLAB® R2014b
31 CommentsOldest to Newest
Thanks Andy: this is a fantastic article, and I’m really looking forward to this blog.
I was inspired by the article to try setting up Jenkins and to hook it up to the unit test suite I have in MATLAB. However, I’m on Windows, and since Jenkins is a little more focused on Unix environments there were a couple of issues. But you’ll be glad to hear I’ve now solved them, so for the benefit of other readers who might want to try the same thing, here are a few notes.
i) When you add a build step as “Execute shell”, Jenkins attempts to execute your build step using the shell command sh. This is a Unix command that is not typically present on Windows. The Jenkins documentation recommends that Windows users should install the free software UnxUtils, which provides a Windows implementation of sh; however I have not been able to make this approach work.
ii) Nevertheless, adding build steps as “Execute Windows batch command” works fine – with one wrinkle.
iii) The wrinkle is that Jenkins then wraps the build step in a Windows batch file, and executes it with a call to cmd.exe. Now if you specify your MATLAB build step as Andy did in the article (i.e. matlab -nodisplay -r “runALL_THE_TESTS”), the call to cmd.exe will call the MATLAB command and then immediately exit with a successful exit code. Thus all your tests will appear to pass. In addition, if you’re using a TAP plugin (as described in Andy’s next article), Jenkins will proceed to attempt to analyse the TAP file before MATLAB has had a chance to create it.
iv) The solution, I’ve found, is to add the -wait option to the MATLAB build step (i.e. it is now matlab -nodisplay -wait -r “runALL_THE_TESTS”). This prevents the call to cmd.exe from exiting before MATLAB has finished; it exits with the proper exit code as supplied by MATLAB and reports this to Jenkins; and Jenkins does not attempt to analyse any TAP file before it has been created.
With those modifications to the process outlined in Andy’s article, I now have a sweet CI system up and running to automate all my MATLAB build and test needs – and I should say that other than the above wrinkles, it’s been a breeze to get set up. Hope this helps anyone else who’d like to try the same thing!
Thanks Sam for posting these Windows specific caveats, it is very helpful!
I have investigated using Jenkins with Matlab in a Windows environment.. It is very useful to retrieve the data from Git. It is very easy to setup in Jenkins.
We found that for accessing network-drives you have to cross the security obstacles. When running Jenkins as a service it is difficult to access the network drives as the mappings (eg.g G:\) are not known. Another issue is logging in as a user for the Jenkins service.
I found it is far more easy to run Jenkins in a WAR container (so in a Java sandbox)! This will solve any issues with Windows file network access.
Thanks for the info Han. Just curious, did you look into using UNC paths to avoid the drive letter issues?
yes, I tried to use UNC paths but this didn’t work.In fact this was my first try to solve the issue.
Andy, very excited to see this new blog in general and this article on Jenkins in particular.
I’m looking forward to learning more about the tools and processes that you guys use to develop MATLAB , that can also help us as users. Thanks in advance.
Thanks for the excitement Ben! Definitely hope to hear from users like you to hear what others in the community are doing as well.
One question though: Are there any licensing issues with using Jenkins with Matlab? It seems that the Matlab license is sort of restrictive when it comes to remote usage etc. (It would be really great if there were no obstacles for CI, it’s such a powerful tool.)
I am not really the best person to talk to about licensing questions since licensing is not my expertise. I would suggest having a conversation with your account manager to get a good understanding of the licensing terms and how they might relate to a continuous integration workflow.
thanks for the great article. Do you know if there is a possibilty to integrate the Profile Report from Matlab in Jenkins? For the Code Coverage I used a work around with the Cobertura Plugin for Jenkins and the Coverage Report Generator from Jonathan Karr (http://www.mathworks.com/matlabcentral/fileexchange/33972-coverage-report-generator). But the execution Time would be interesting too.
Thanks for sharing what you are doing, it is very interesting to hear your use of the Cobertura plugin. We can definitely look into ways to make other reports such as profiling data available in a CI context. Quick question, are you looking for the profiler data because you are interested in tracking code performance over time? Is this really speaking to a need for some performance testing infrastructure?
Thanks for your fast response. Primary we want to see the coverage of our unit tests, the static Warnings (analyzed by checkcode) and the Warnings during runtime in jenkins (all works now). For now we thougt that it would be great to see the report of the profiler in jenkins too. Currently I try to implement it with the HTML Publisher Plugin in Jenkins and the save as HTML option of the profiler. It is a bit complicated because the resources like the .css or image files are linked as absolute path to the installation of matlab. When publishing the report on jenkins all formatting is gone. Moreover the reports cover a lot of files we are not interessted in (for example the class ‘TestRunner’). Mostly we are interested only in our self written packages. I know there is a ‘Details’ parameter for the profiler but there are hardly any options.
Back to your question: For now the tracking of code performance over time is not that important to us. But that is certainly related to the cost of implementation. At the moment it is sufficient to visualize the current performance.
Im using CircleCi as our CI system. CircleCI is a cloud base Continuous Integration system, which downloads a dedicated docker iamge for each build and runs ours tests on it. This means that we should strive to have a minimal size docker image in order to have a fast build. Installing a ~10GB application not an option with CircleCI architecture because downloading an image will take longer than running the tests.
What is the minimal matlab installation that is needed in order to perform Continuous Integration with Matlab? MCR is an option but with that I can’t run matlab like /Applications/MATLAB_R2014b.app/bin/matlab -nodisplay -r “runALL_THE_TESTS”.
I had some additional problems with Jenkins on Windows using 2015.a, and ended up with the following solution working from Jenkins on Windows:
RunAllUnitTests.m (file must be in the root folder of the workspace):
%Add subdirectories to Matlab path, to allow tests to run if you have a directory structure
%Create and run the test suite
suiteFolder = TestSuite.fromFolder([pwd ‘\Tests\UnitTests’]);
results = run(suiteFolder);
In Jenkins, add “Execute Windows batch command”, paste in the following. I needed to output to the text file and then store the errorlevel from the Matlab call because the console output was not being piped to stdout, this may be a Windows limitation or possibly something in my specific environment:
start /wait matlab -nodesktop -nosplash -minimize -wait -r “RunAllUnitTests” -logfile unittestlog.txt
By calling EXIT and using the output value of the Matlab call, it will cause the build to fail if any tests fail. Hope this helps.
@Nir – my apologies for missing your question. It seems that CircleCI does not support MATLAB unfortunately. Do you know if they have any self-hosted options for the CI system?
@Guy – Thanks for posting this Windows specific recipe. It no doubt will be helpful for others with CI systems running windows. Also, hopefully you’ve seen some other CI related posted in case you want better CI integration:
thanks for this very usefull article.
I try to set up this solution in my windows environment and I meet some difficulties: The result of the matlab execution is not displayed on jenkins consol, even for a simple “hello world”.
I have tried the solution given by Guy, but jenkins seems to be blocked by the first instruction of the script:
start /wait matlab -nodesktop -nosplash -minimize -wait -r “RunAllUnitTests” -logfile unittestlog.txt
Does anyone can help me?
Thanks by advance,
I used the following under ‘Execute Windows batch command’ option in Jenkins
matlab -nodisplay -nosplash -nodesktop -wait -r “runMyTests(”, true)”
Where runMyTests basically takes a true / false parameter to tell it about whether or not it’s jenkins. You use that parameter to control the exit behaviour, which informs Jenkins whether or not it finished cleanly or otherwise…
@Simon: Thanks for your quick answer. Could you please describe how you handle the “true/false” parameter inside your runMyTests bash script? I don’t get the difference with a simple exit command.
here is my script:
suite = TestSuite.fromFile(‘AverageTest.m’);
results = run(suite);
I finally manage to get the matlab outup in the Jenkins console by following Guy Starbuck tips. I have to fix two errors:
1- Set the proper working directory in Jenkins
2- Change the quote from Guy commands from curly quotes to straight one.
Thanks for your help.
Hi Andy, thanks for the informative write up! Unfortunately I am using Windows and have experienced an issue getting Jenkins to run my unit tests. I have copied all files to the working directory, and using the command line in the working directory can invoke my test script with
matlab -nosplash -wait -r “myTests” -logfile unittestlog.txt
However, when I run a build with Jenkins using the above command as a Windows batch command the build will run indefinitely. When I look in console output I see the call to matlab and then a spinning circle until I abort the build. If I remove -wait the execution completes immediately and is reported as a success. I have tried using Guy Starbuck’s method, however I did not understand what Thomas meant by replacing ‘curly quotes to straight one’, so maybe I missed something?
Any suggestions would be much appreciated!!
Also, I am running 2015.b
Tough to tell what is going on without a bit more info. What are the contents of your “myTests” function/script?
Guy’s curly quotes issue stemmed from copying content from Microsoft Word where standard straight double quotes are replaced by curly quotes and this was causing some problems when these quotes were copy/pasted into the command.
Thanks for the reply Andy, it turns out Jenkins was using the ‘local services’ user instead of my network user credentials which was causing our license server to reject a license request for the Jenkins service’s MATLAB instance. I’m not really sure how all that works for corporate environments, but long story short I had to go to Start -> Administrative Tools -> Services, scroll to Jenkins, right click -> Properties -> LongOn Tab, and then search for my user credentials on the network. Now it’s all good!
Just as a note, I found the following Stack Exchange post helpful in figuring out how to troubleshoot MATLAB issues when using Jenkins, particularly the second answer and replies (I’m assuming you’re the same Andy Campbell from some of the replies :-D)
we have been very sucesfully using Jenkins with Matlab for running unittests per checkin and daily tests. We would like to also generate reports of code coverage. There is a cobutera plugin that someone wrote on file exchange but it doesnt really work great. are there any plans to support cpde coverage with Jenkins?
This is something that many of you want and we are actively investigating. Are you tied to Cobertura specifically or are you more interested in making sure that we can integrate coverage reports into your CI workflow in some form?
right now I am investigating to get the results of the profiler into Jenkins by using Cobertura XML format.
My use case is to get the coverage of code under test when calling a collection of tests (TestSuite).
You wrote that you’re actively investigating this, any news on it?
any news on the status of your investigation? I am very interested in a Cobertura producing plugin.
We are definitely still working on it and am interested in any information about use cases people are running into and specifically what people are looking for. For example, we are interested to hear about whether (and why) Cobertura is the preferred format over others and which CI systems are being used. Do you have any insight?
I am not able to establish connection between Jenkins & MATLAB.
I have installed jenkins executable from website & then tried steps given by you, but its not working out.
I tried with tomcat server & jenkins.war file setup as well, still no progress as suggested by some other forums.
I could not succeed in opening a simple application like Notepad.
This sounds like your Jenkins is somehow misconfigured. I would suggest taking a look at the Jenkins documentation (it can be found at the Jenkins site). If you don’t have any luck, perhaps you can try the Jenkins user’s mailing list.
I tried already tried installing jenkins multiple times. This is my console window of Jenkins
Started by user Chaitanya Kulkarni
Building in workspace C:\Windows\system32\config\systemprofile\.jenkins\workspace\Test_1
[Test_1] $ cmd /c call D:\Public\Temp\hudson5022438301719740064.bat
C:\>matlab -nodisplay -r “disp(‘Hello World!’);exit”
Its giving message as success, but not establishing any connection with it. I’m not getting any MATLAB information on the console window.
My question is can we invoke MATLAB.exe from jenkins to open in different window itself.?
My issue is solved!! Seems that there was some some issue with installation only. I am using jenkins with tomcat server & war file.