{"id":7,"date":"2015-01-20T17:34:40","date_gmt":"2015-01-20T17:34:40","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=7"},"modified":"2015-01-21T15:39:49","modified_gmt":"2015-01-21T15:39:49","slug":"the-other-kind-of-continuous-integration","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2015\/01\/20\/the-other-kind-of-continuous-integration\/","title":{"rendered":"The Other Kind of Continuous Integration"},"content":{"rendered":"\r\n\r\n

Hi there. I am super excited to introduce our new Developer Zone blog! I work in a group at MathWorks that is responsible for building the infrastructure and architecture that we require as a company to deliver software that is of high quality, on schedule, and meets customer requirements. In short, we develop the build, test, and integration tools needed to produce production grade software on multiple platforms and using many different technologies. My specific team is focused on test infrastructure and are the primary developers of MATLAB's unit test framework<\/a>.<\/p>

As you might imagine I am very interested in development approaches and what it takes to build production software. This blog is a recognition of the fact that we are not the only ones who have a passion for this. Indeed, we hear from many of you who are facing similar tasks building production grade software using MATLAB® as a critical piece of the technology stack, and we have literally thousands of world class MATLAB developers in house doing the same thing. I am excited to share some of the building blocks we use to develop robust, extendable, flexible software that is written in MATLAB, and I am likewise excited to hear how you are doing the same.<\/p>

To start, for our blog's first post I thought it would be fitting to describe how MATLAB can interface with a CI system which is a must have for collaborating with others to build high quality software.<\/p>

CI? C-who?<\/b><\/p>

Wait, what am I talking about, CI system? Well, what do you think of when you hear the term \"Continuous Integration\"? Do you think of calculating anti-derivatives and definite integrals of continuous functions or interfacing with a system that helps build and test software? A quick google check shows that most people think the latter, although perhaps those involved in mathematics may skew differently.<\/p>

Whatever the case may be, developing software using a continuous integration (CI) system<\/a> has proven to be essential for production grade software. The services and safety net that these systems provide enable the creation of software that can rise to high levels of safety and quality.<\/p>

So how is this done using MATLAB?<\/p>

There are a few common purposes of continuous integration systems. Three important purposes are:<\/p>

  1. Interfacing with a source control system<\/li>
  2. Building software that requires a compilation step like C++ or Java®<\/li>
  3. Testing software that has changed or been rebuilt<\/li><\/ol><\/div>

    Connecting your CI system with your source control system depends on both systems and this can be done independently from MATLAB.<\/p>

    As for building software, there are many features of MATLAB which do indeed require a build step, such as writing MEX files, writing Java classes that work with MATLAB, or p-coding MATLAB files for distribution. However, probably the most common MATLAB activity is writing code using the MATLAB language, which does not require a build step. For now, let's scope this discussion to these types of MATLAB activities that do not have any artifacts to build (i.e. the MATLAB code itself is the artifact).<\/p>

    Connect MATLAB<\/b><\/p>

    If you are producing and sharing MATLAB code files, and they are already configured in your source control system then great, we are almost there. We just need to make sure that when you change your MATLAB source code your tests get run in your CI System and trigger build failures. Well, that is what I want to explore in more depth today. Note that there are a variety of CI systems out there<\/a>, but let's look at connecting MATLAB tests to Jenkins™<\/a>, which is popular and integrates with a variety of systems.<\/p>

    To do this we need to:<\/p>

    1. Configure Jenkins to launch MATLAB, run the tests, and close MATLAB<\/li>
    2. Communicate the results of the test run from MATLAB to the Jenkins software<\/li>
    3. Ride out our days relaxing on the beach instead of manually integrating our code changes<\/li><\/ol><\/div>

      Launching MATLAB can be done by creating a new free-form project in Jenkins. This type of project will allow us to set up a simple build script to launch MATLAB and run the tests. In this case I've named the job \"Run ALL THE TESTS\".<\/a><\/p>

      \"\" <\/p>

      Now we can create our build action to run a shell script.<\/p>

      \"\" <\/p>

      The shell command that we use needs to simply start MATLAB and run something. We can use this simple shell command to just confirm we can get this MATLAB running. Use this script and run a test build in Jenkins:<\/p>

        \/Applications\/MATLAB_R2014b.app\/bin\/matlab -nodisplay -r \"disp('Hello World!');exit\"<\/pre>
      \"\" <\/p>
      Connect the Tests!<\/b><\/p>
      Great! We are connected. Next step is to actually run something in MATLAB. This is dependent upon on how you have your tests organized and precisely which tests you wish to run. This exercise demonstrates running tests that are located in a \"testcases\" package as well as its sub-packages. Here's the updated shell script:<\/p>
      \/Applications\/MATLAB_R2014b.app\/bin\/matlab -nodisplay -r \"runALL_THE_TESTS\"\r\n<\/pre>
      and here is the content of the runAll_THE_TESTS<\/tt> script:<\/p>
      import matlab.unittest.TestSuite<\/span>;\r\n\r\ntry<\/span>\r\n    suite = TestSuite.fromPackage('testcases'<\/span>,'IncludingSubpackages'<\/span>,true);\r\n    results = run(suite);\r\n    display(results);\r\ncatch<\/span> e\r\n    disp(getReport(e,'extended'<\/span>));\r\n    exit(1);\r\nend<\/span>\r\nexit;\r\n<\/pre>
      So in short we want to create a TestSuite from our package and its sub-packages and then just run the tests using the default TestRunner. The try-catch usage is to allow MATLAB to exit quickly in the event of some unforeseen problem, but notice that before doing so the code prints the extended error details and exits with a non-zero status code. This will ensure that the Jenkins task will cause a build failure if something goes terribly wrong.<\/p>
      Let's run another job and see our tests in action<\/p>
      \"\" <\/p>
      We are in the business of running tests! Unfortunately we are only running them and not reacting in any way to test failures. You can see from the blue job marker above that the Jenkins job actually passed despite many of the tests actually failing. This is no good! Looking at the bottom of the test log we see that 163 tests passed, 70 tests failed, and 80 tests did not complete, so why did the job pass?<\/p>
      \"\" <\/p>
      What is happening here is that the Jenkins CI system does not know the results of this test run. The job determined pass or failure by looking at the exit status code of the MATLAB process we kicked off. While the tests failed, MATLAB was entirely happy as it started, ran the failing test suite and then exited cleanly. We need to tell MATLAB to exit with a non-zero status code if any test failures were encountered. We can do this by simply changing the last line of our run script to do just that:<\/p>
        exit(any([results.Failed]));<\/pre>
      Running the job we can see that we now see the test failures as job failures<\/p>
      \"\" <\/p>
      \"\" <\/p>
      For the win!<\/b><\/p>
      After just a few minutes we are now in business with MATLAB connected to our Jenkins CI system, running tests and failing jobs when those tests fail. Awesome! This will work in all releases that support the test framework (all the way back to R2013a), and now you can set up Jenkins run these builds on a regular basis, such as nightly or whenever a file is checked into the SCM system. If you have R2014a or later we can do even better, but I'll save that for the next blog post.<\/p>
      Do you automatically test your MATLAB code changes with a CI system? If so, which one? What benefits have you realized in your development process? What tips and tricks have you learned along the way? Let us know in the comments section below.<\/p>