Steven Lord, Andy Campbell, and David Hruska are members of the Quality Engineering group at MathWorks who are guest blogging today to introduce a new feature in R2013a, the MATLAB unit testing infrastructure. There are several submissions on the MATLAB Central File Exchange related to unit testing of MATLAB code. Blogger Steve Eddins wrote one highly rated example back in 2009. In release R2013a, MathWorks included in MATLAB itself a MATLAB implementation of the industry-standard xUnit testing framework.
If you're not a software developer, you may be wondering if this feature will be of any use to you. In this post, we will describe one way someone who may not consider themselves a software developer may be able to take advantage of this framework using the example of a professor grading students' homework submissions. That's not to say that the developers in the audience should move on to the next post; you can use these tools to test your own code just like a professor can use them to test code written by his or her students.
There is a great deal of functionality in this feature that we will not show here. For more information we refer you to the MATLAB Unit Testing Framework documentation.
In order to use this feature, you should be aware of how to define simple MATLAB classes in classdef files, how to define a class that inherits from another, and how to specify attributes for methods and properties of those classes. The object-oriented programming documentation describes these capabilities.
As a professor in an introductory programming class, you want your students to write a program to compute Fibonacci numbers. The exact problem statement you give the students is:
Create a function "fib" that accepts a nonnegative integer n and returns the nth Fibonacci number. The Fibonacci numbers are generated by this relationship:
F(0) = 1 F(1) = 1 F(n) = F(n-1) + F(n-2) for integer n > 1
Your function should throw an error if n is not a nonnegative integer.
The most basic MATLAB unit test is a MATLAB classdef class file that inherits from the matlab.unittest.TestCase class. Throughout the rest of this post we will add additional pieces to this basic framework to increase the capability of this test and will change its name to reflect its increased functionality.
1 classdef basicTest < matlab.unittest.TestCase 2 3 end
test = basicTest
test = basicTest with no properties.
To run the test, we can simply pass test to the run function. There are more advanced ways that make it easier to run a group of tests, but for our purposes (checking one student's answer at a time) this will be sufficient. When you move to checking multiple students' answers at a time, you can use run inside a for loop.
Since basicTest doesn't actually validate the output from the student's function, it doesn't take very long to execute.
results = run(test)
results = 0x0 TestResult array with properties: Name Passed Failed Incomplete Duration Totals: 0 Passed, 0 Failed, 0 Incomplete. 0 seconds testing time.
Let's say that a student named Thomas submitted a function fib.m as his solution to this assignment. Thomas's code is stored in a sub-folder named thomas. To set up our test to check Thomas's answer, we add the folder holding his code to the path.
addpath('thomas'); dbtype fib.m
1 function y = fib(n) 2 if n <= 1 3 y = 1; 4 else 5 y = fib(n-1)+fib(n-2); 6 end
The basicTest is a valid test class, and we can run it, but it doesn't actually perform any validation of the student's test file. The methods that will perform that validation need to be written in a methods block that has the attribute Test specified.
The matlab.unittest.TestCase class includes qualification methods that you can use to test various qualities of the results returned by the student files. The qualification method that you will likely use most frequently is the verifyEqual method, which passes if the two values you pass into it are equal and reports a test failure if they are not.
The documentation for the matlab.unittest.TestCase class lists many other qualification methods that you can use to perform other types of validation, including testing the data type and size of the results; matching a string result to an expected string; testing that a given section of code throws a specific errors or issues a specific warning; and many more.
This simple test builds upon generalTest by adding a test method that checks that the student's function returns the value 1 when called with the input 0.
1 classdef simpleTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 end 9 end
Thomas's solution to the assignment satisfies this basic check. We can use the results returned from run to display the percentage of the tests that pass.
results = run(simpleTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']);
Running simpleTest . Done simpleTest __________ results = TestResult with properties: Name: 'simpleTest/fibonacciOfZeroShouldBeOne' Passed: 1 Failed: 0 Incomplete: 0 Duration: 0.0112 Totals: 1 Passed, 0 Failed, 0 Incomplete. 0.011168 seconds testing time. 100% Passed.
Now that we have a basic positive test in place we can add in a test that checks the behavior of the student's function when passed a non-integer value (like n = pi) as input. The assignment stated that when called with a non-integer value, the student's function should error. Since the assignment doesn't require a specific error to be thrown, the test passes as long as fib(pi) throws any exception.
1 classdef errorCaseTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 function fibonacciOfNonintegerShouldError(testCase) 9 testCase.verifyError(@()fib(pi), ?MException); 10 end 11 end 12 end
Thomas forgot to include a check for a non-integer valued input in his function, so our test should indicate that by reporting a failure.
results = run(errorCaseTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']);
Running errorCaseTest . ================================================================================ Verification failed in errorCaseTest/fibonacciOfNonintegerShouldError. --------------------- Framework Diagnostic: --------------------- verifyError failed. --> The function did not throw any exception. Expected Exception Type: MException Evaluated Function: @()fib(pi) ------------------ Stack Information: ------------------ In C:\Program Files\MATLAB\R2013a\toolbox\matlab\testframework\+matlab\+unittest\+qualifications\Verifiable.m (Verifiable.verifyError) at 637 In H:\Documents\LOREN\MyJob\Art of MATLAB\errorCaseTest.m (errorCaseTest.fibonacciOfNonintegerShouldError) at 9 ================================================================================ . Done errorCaseTest __________ Failure Summary: Name Failed Incomplete Reason(s) ============================================================================================= errorCaseTest/fibonacciOfNonintegerShouldError X Failed by verification. results = 1x2 TestResult array with properties: Name Passed Failed Incomplete Duration Totals: 1 Passed, 1 Failed, 0 Incomplete. 0.026224 seconds testing time. 50% Passed.
Another student, Benjamin, checked for a non-integer value in his code as you can see on line 2.
rmpath('thomas'); addpath('benjamin'); dbtype fib.m
1 function y = fib(n) 2 if (n ~= round(n)) || n < 0 3 error('N is not an integer!'); 4 elseif n == 0 || n == 1 5 y = 1; 6 else 7 y = fib(n-1)+fib(n-2); 8 end
Benjamin's code passed both the test implemented in the fibonacciOfZeroShouldBeOne method (which we copied into errorCaseTest from simpleTest) and the new test case implemented in the fibonacciOfNonintegerShouldError method.
results = run(errorCaseTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']);
Running errorCaseTest .. Done errorCaseTest __________ results = 1x2 TestResult array with properties: Name Passed Failed Incomplete Duration Totals: 2 Passed, 0 Failed, 0 Incomplete. 0.010132 seconds testing time. 100% Passed.
The problem statement given earlier in this post is a plain text description of the homework assignment we assigned to the students. We can also state the problem for the students in code (if they're using release R2013a or later) by giving them a test file they can run just like simpleTest or errorCaseTest. They can directly use this "requirement test" to ensure their functions satisfy the requirements of the assignment.
1 classdef studentTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 function fibonacciOfNonintegerShouldError(testCase) 9 testCase.verifyError(@()fib(pi), ?MException); 10 end 11 end 12 end
In order for the student's code to pass the assignment, it will need to pass the test cases given in the studentTest unit test. However, we don't want to use studentTest as the only check of the student's code. If we did, the student could write their function to cover only the test cases in the student test file.
We could solve this problem by having two separate test files, one containing the student test cases and one containing additional test cases the instructor uses in the grading process. Can we avoid having to run both test files manually or duplicating the code from the student test cases in the instructor test? Yes!
To do so, we write an instructor test file to incorporate, through inheritance, the student test file. We can then add additional test cases to the instructor test file. When we run this test it should run three test cases; two inherited from studentTest, fibonacciOfZeroShouldBeOne and fibonacciOfNonintegerShouldError, and one from instructorTest itself, fibonacciOf5.
1 classdef instructorTest < studentTest 2 % Because the student test file is a matlab.unittest.TestCase and 3 % instructorTest inherits from it, instructorTest is also a 4 % matlab.unittest.TestCase. 5 6 methods(Test) 7 function fibonacciOf5(testCase) 8 % Evaluate the student's function for n = 5 9 result = fib(5); 10 testCase.verifyEqual(result, 8, 'Fibonacci(5) should be 8'); 11 end 12 end 13 end
Let's look at Eric's test file that passes the studentTestFile test, but in which he completely forgot to implement the F(n) = F(n-1)+F(n-2) recursion step.
rmpath('benjamin'); addpath('eric'); dbtype fib.m
1 function y = fib(n) 2 if (n ~= round(n)) || n < 0 3 error('N is not an integer!'); 4 end 5 y = 1;
It should pass the student unit test.
results = run(studentTest); percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']);
Running studentTest .. Done studentTest __________ 100% Passed.
It does NOT pass the instructor unit test because it fails one of the test cases.
results = run(instructorTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']);
Running instructorTest .. ================================================================================ Verification failed in instructorTest/fibonacciOf5. ---------------- Test Diagnostic: ---------------- Fibonacci(5) should be 8 --------------------- Framework Diagnostic: --------------------- verifyEqual failed. --> NumericComparator failed. --> The values are not equal using "isequaln". Actual Value: 1 Expected Value: 8 ------------------ Stack Information: ------------------ In C:\Program Files\MATLAB\R2013a\toolbox\matlab\testframework\+matlab\+unittest\+qualifications\Verifiable.m (Verifiable.verifyEqual) at 411 In H:\Documents\LOREN\MyJob\Art of MATLAB\instructorTest.m (instructorTest.fibonacciOf5) at 10 ================================================================================ . Done instructorTest __________ Failure Summary: Name Failed Incomplete Reason(s) ========================================================================== instructorTest/fibonacciOf5 X Failed by verification. results = 1x3 TestResult array with properties: Name Passed Failed Incomplete Duration Totals: 2 Passed, 1 Failed, 0 Incomplete. 0.028906 seconds testing time. 66.6667% Passed.
Benjamin, whose code we tested above, wrote a correct solution to the homework problem.
rmpath('eric'); addpath('benjamin'); results = run(instructorTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']); rmpath('benjamin');
Running instructorTest ... Done instructorTest __________ results = 1x3 TestResult array with properties: Name Passed Failed Incomplete Duration Totals: 3 Passed, 0 Failed, 0 Incomplete. 0.015946 seconds testing time. 100% Passed.
In this post, we showed you the basics of using the new MATLAB unit testing infrastructure using homework grading as a use case.
We checked that the student's code worked (by returning the correct answer) for one valid value and worked (by throwing an error) for one invalid value. We also showed how you can use this infrastructure to provide an aid/check for the students that you can also use as part of your grading.
We hope this brief introduction to the unit testing framework has shown you how you can make use of this feature even if you don't consider yourself a software developer. Let us know in the comments for this post how you might use this new functionality. Or, if you've already tried using matlab.unittest, let us know about your experiences here.
Published with MATLAB® R2013a
Comments are closed.
8 CommentsOldest to Newest
I think it’s fantastic that matlab has its own built-in xUnit framework.
I do quite a bit of matlab OO programming in a daily basis, and one of the most exasperating issues is the verbosity of the matlab language in these matters (i.e. property block, static methods block, regular methods block …).
In the documentation of the new framework explains that in order to define the setUp and tearDown methods the actual code should be like:
testCase.TestFigure = figure;
whereas in the framework linked suffices a method called setUp(self) and it is execute on a name convection basis. Less code to write, less time to actually solve the problem.
Anyway, it is a good thing have a built-in support for unit testing.
Thanks for taking a look at the framework!
I understand your preference for limiting the verbosity. For this interface we have weighed the simplicity of naming conventions against the robustness and flexibility of defining them with explicit constructs. We have landed on the latter approach. This has a number of benefits including:
* Decoupling the name of a method from how the method is intended to be used by the framework.
* Enabling using test base classes that set up fixtures needed for the base class and all subclasses. With this interface there is no need to override the “setUp” method in a subclass and remember to call through to the base class setUp method. Rather, the base class can simply define its own fixture method(s), even making them Sealed so they cannot be overridden, and the subclass can create its own independent method for the portion of the setUp needed by the subclass.
* Allowing more familiarity with the more modern xUnit implementations such as JUnit 4 (rather than JUnit 3) and NUnit which have evolved away from naming convention and toward annotation/attribute style approaches.
* Preventing the enforcement of a naming convention. I am all for “convention over configuration” approaches, but this requires that there be some way to opt out of the convention if desired. However, relying so explicitly on the naming convention by the framework precludes the ability for someone to choose not to adhere, or perhaps choose a *different* naming convention to fit inside their organizational guidelines. We felt the decision of what to name methods and classes should not be ours but rather the test writer using the interface, and this approach supports that.
* Logically grouping methods by their purpose. While it is indeed more verbose, there is a benefit in separating the “TestMethodSetup” methods from the “Test” methods from even the normal helper methods that are not used directly by the framework. Keeping them separated into their own methods blocks tends to keep code similar in function close to one another, having a side benefits of actually making it easier to find the method of interest at times.
Anyway, thank you again for taking a look and I encourage you to take a closer look at many other benefits and features in this new framework including:
* The rich library of qualification (assertion) methods available
* The ability to create fixtures that are shared between all the methods of a test class
* The increased diagnostics infrastructure which does a better job of describing the problem encountered when a failure occurs
* The full support of the framework by The MathWorks
* The knowledge that tests can be written that can be run and shared with colleagues without any additional download or setup as long as they have R2013a or later.
…and please if you do continue to evaluate it, keep the feedback coming!
Do the TearDown methods execute even if the test code errors out or the user Ctrl-C’s (like an onCleanup)?
Yes, the code defined in the teardown methods executes in an exception safe way.
There is also another feature that allows adding teardown code dynamically and executes in a deterministic order (Last-In-First-Out, or LIFO). Look here for more information:
Eric: yes, setup and teardown are handled in an exception-safe manner so even if the test execution is terminated for some reason, the teardown routines will still execute.
Thanks, that info could be handy to add to the doc if it isn’t there already. Cheers
Is there a way to include a timeout for this kind of unit testing? For example, if the student includes an infinite while loop, is there a way to automatically fail the test case after 5 seconds?
@David: Unfortunately, the MATLAB language doesn’t currently provide a way to programmatically terminate execution of code after a certain period of time. I’ve submitted this suggestion as an enhancement request.