Using the MATLAB Unit Testing Infrastructure for Grading Assignments
Steven Lord, Andy Campbell, and David Hruska are members of the Quality Engineering group at MathWorks who are guest blogging today to introduce a new feature in R2013a, the MATLAB unit testing infrastructure. There are several submissions on the MATLAB Central File Exchange related to unit testing of MATLAB code. Blogger Steve Eddins wrote one highly rated example back in 2009. In release R2013a, MathWorks included in MATLAB itself a MATLAB implementation of the industry-standard xUnit testing framework.
If you're not a software developer, you may be wondering if this feature will be of any use to you. In this post, we will describe one way someone who may not consider themselves a software developer may be able to take advantage of this framework using the example of a professor grading students' homework submissions. That's not to say that the developers in the audience should move on to the next post; you can use these tools to test your own code just like a professor can use them to test code written by his or her students.
There is a great deal of functionality in this feature that we will not show here. For more information we refer you to the MATLAB Unit Testing Framework documentation.
Contents
Background
In order to use this feature, you should be aware of how to define simple MATLAB classes in classdef files, how to define a class that inherits from another, and how to specify attributes for methods and properties of those classes. The object-oriented programming documentation describes these capabilities.
Problem Statement
As a professor in an introductory programming class, you want your students to write a program to compute Fibonacci numbers. The exact problem statement you give the students is:
Create a function "fib" that accepts a nonnegative integer n and returns the nth Fibonacci number. The Fibonacci numbers are generated by this relationship:
F(0) = 1 F(1) = 1 F(n) = F(n-1) + F(n-2) for integer n > 1
Your function should throw an error if n is not a nonnegative integer.
Basic Unit Test
The most basic MATLAB unit test is a MATLAB classdef class file that inherits from the matlab.unittest.TestCase class. Throughout the rest of this post we will add additional pieces to this basic framework to increase the capability of this test and will change its name to reflect its increased functionality.
dbtype basicTest.m
1 classdef basicTest < matlab.unittest.TestCase 2 3 end
test = basicTest
test = basicTest with no properties.
Running a Test
To run the test, we can simply pass test to the run function. There are more advanced ways that make it easier to run a group of tests, but for our purposes (checking one student's answer at a time) this will be sufficient. When you move to checking multiple students' answers at a time, you can use run inside a for loop.
Since basicTest doesn't actually validate the output from the student's function, it doesn't take very long to execute.
results = run(test)
results =
0x0 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Totals:
0 Passed, 0 Failed, 0 Incomplete.
0 seconds testing time.
Let's say that a student named Thomas submitted a function fib.m as his solution to this assignment. Thomas's code is stored in a sub-folder named thomas. To set up our test to check Thomas's answer, we add the folder holding his code to the path.
addpath('thomas'); dbtype fib.m
1 function y = fib(n) 2 if n <= 1 3 y = 1; 4 else 5 y = fib(n-1)+fib(n-2); 6 end
Test that F(0) Equals 1
The basicTest is a valid test class, and we can run it, but it doesn't actually perform any validation of the student's test file. The methods that will perform that validation need to be written in a methods block that has the attribute Test specified.
The matlab.unittest.TestCase class includes qualification methods that you can use to test various qualities of the results returned by the student files. The qualification method that you will likely use most frequently is the verifyEqual method, which passes if the two values you pass into it are equal and reports a test failure if they are not.
The documentation for the matlab.unittest.TestCase class lists many other qualification methods that you can use to perform other types of validation, including testing the data type and size of the results; matching a string result to an expected string; testing that a given section of code throws a specific errors or issues a specific warning; and many more.
This simple test builds upon generalTest by adding a test method that checks that the student's function returns the value 1 when called with the input 0.
dbtype simpleTest.m
1 classdef simpleTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 end 9 end
Thomas's solution to the assignment satisfies this basic check. We can use the results returned from run to display the percentage of the tests that pass.
results = run(simpleTest)
percentPassed = 100 * nnz([results.Passed]) / numel(results);
disp([num2str(percentPassed), '% Passed.']);
Running simpleTest
.
Done simpleTest
__________
results =
TestResult with properties:
Name: 'simpleTest/fibonacciOfZeroShouldBeOne'
Passed: 1
Failed: 0
Incomplete: 0
Duration: 0.0112
Totals:
1 Passed, 0 Failed, 0 Incomplete.
0.011168 seconds testing time.
100% Passed.
Test that F(pi) Throws an Error
Now that we have a basic positive test in place we can add in a test that checks the behavior of the student's function when passed a non-integer value (like n = pi) as input. The assignment stated that when called with a non-integer value, the student's function should error. Since the assignment doesn't require a specific error to be thrown, the test passes as long as fib(pi) throws any exception.
dbtype errorCaseTest.m
1 classdef errorCaseTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 function fibonacciOfNonintegerShouldError(testCase) 9 testCase.verifyError(@()fib(pi), ?MException); 10 end 11 end 12 end
Thomas forgot to include a check for a non-integer valued input in his function, so our test should indicate that by reporting a failure.
results = run(errorCaseTest)
percentPassed = 100 * nnz([results.Passed]) / numel(results);
disp([num2str(percentPassed), '% Passed.']);
Running errorCaseTest
.
================================================================================
Verification failed in errorCaseTest/fibonacciOfNonintegerShouldError.
---------------------
Framework Diagnostic:
---------------------
verifyError failed.
--> The function did not throw any exception.
Expected Exception Type:
MException
Evaluated Function:
@()fib(pi)
------------------
Stack Information:
------------------
In C:\Program Files\MATLAB\R2013a\toolbox\matlab\testframework\+matlab\+unittest\+qualifications\Verifiable.m (Verifiable.verifyError) at 637
In H:\Documents\LOREN\MyJob\Art of MATLAB\errorCaseTest.m (errorCaseTest.fibonacciOfNonintegerShouldError) at 9
================================================================================
.
Done errorCaseTest
__________
Failure Summary:
Name Failed Incomplete Reason(s)
=============================================================================================
errorCaseTest/fibonacciOfNonintegerShouldError X Failed by verification.
results =
1x2 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Totals:
1 Passed, 1 Failed, 0 Incomplete.
0.026224 seconds testing time.
50% Passed.
Another student, Benjamin, checked for a non-integer value in his code as you can see on line 2.
rmpath('thomas'); addpath('benjamin'); dbtype fib.m
1 function y = fib(n)
2 if (n ~= round(n)) || n < 0
3 error('N is not an integer!');
4 elseif n == 0 || n == 1
5 y = 1;
6 else
7 y = fib(n-1)+fib(n-2);
8 end
Benjamin's code passed both the test implemented in the fibonacciOfZeroShouldBeOne method (which we copied into errorCaseTest from simpleTest) and the new test case implemented in the fibonacciOfNonintegerShouldError method.
results = run(errorCaseTest)
percentPassed = 100 * nnz([results.Passed]) / numel(results);
disp([num2str(percentPassed), '% Passed.']);
Running errorCaseTest
..
Done errorCaseTest
__________
results =
1x2 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Totals:
2 Passed, 0 Failed, 0 Incomplete.
0.010132 seconds testing time.
100% Passed.
Basic Test for Students, Advanced Tests for Instructor
The problem statement given earlier in this post is a plain text description of the homework assignment we assigned to the students. We can also state the problem for the students in code (if they're using release R2013a or later) by giving them a test file they can run just like simpleTest or errorCaseTest. They can directly use this "requirement test" to ensure their functions satisfy the requirements of the assignment.
dbtype studentTest.m
1 classdef studentTest < matlab.unittest.TestCase 2 methods(Test) 3 function fibonacciOfZeroShouldBeOne(testCase) 4 % Evaluate the student's function for n = 0 5 result = fib(0); 6 testCase.verifyEqual(result, 1); 7 end 8 function fibonacciOfNonintegerShouldError(testCase) 9 testCase.verifyError(@()fib(pi), ?MException); 10 end 11 end 12 end
In order for the student's code to pass the assignment, it will need to pass the test cases given in the studentTest unit test. However, we don't want to use studentTest as the only check of the student's code. If we did, the student could write their function to cover only the test cases in the student test file.
We could solve this problem by having two separate test files, one containing the student test cases and one containing additional test cases the instructor uses in the grading process. Can we avoid having to run both test files manually or duplicating the code from the student test cases in the instructor test? Yes!
To do so, we write an instructor test file to incorporate, through inheritance, the student test file. We can then add additional test cases to the instructor test file. When we run this test it should run three test cases; two inherited from studentTest, fibonacciOfZeroShouldBeOne and fibonacciOfNonintegerShouldError, and one from instructorTest itself, fibonacciOf5.
dbtype instructorTest.m
1 classdef instructorTest < studentTest 2 % Because the student test file is a matlab.unittest.TestCase and 3 % instructorTest inherits from it, instructorTest is also a 4 % matlab.unittest.TestCase. 5 6 methods(Test) 7 function fibonacciOf5(testCase) 8 % Evaluate the student's function for n = 5 9 result = fib(5); 10 testCase.verifyEqual(result, 8, 'Fibonacci(5) should be 8'); 11 end 12 end 13 end
Let's look at Eric's test file that passes the studentTestFile test, but in which he completely forgot to implement the F(n) = F(n-1)+F(n-2) recursion step.
rmpath('benjamin'); addpath('eric'); dbtype fib.m
1 function y = fib(n)
2 if (n ~= round(n)) || n < 0
3 error('N is not an integer!');
4 end
5 y = 1;
It should pass the student unit test.
results = run(studentTest);
percentPassed = 100 * nnz([results.Passed]) / numel(results);
disp([num2str(percentPassed), '% Passed.']);
Running studentTest .. Done studentTest __________ 100% Passed.
It does NOT pass the instructor unit test because it fails one of the test cases.
results = run(instructorTest)
percentPassed = 100 * nnz([results.Passed]) / numel(results);
disp([num2str(percentPassed), '% Passed.']);
Running instructorTest
..
================================================================================
Verification failed in instructorTest/fibonacciOf5.
----------------
Test Diagnostic:
----------------
Fibonacci(5) should be 8
---------------------
Framework Diagnostic:
---------------------
verifyEqual failed.
--> NumericComparator failed.
--> The values are not equal using "isequaln".
Actual Value:
1
Expected Value:
8
------------------
Stack Information:
------------------
In C:\Program Files\MATLAB\R2013a\toolbox\matlab\testframework\+matlab\+unittest\+qualifications\Verifiable.m (Verifiable.verifyEqual) at 411
In H:\Documents\LOREN\MyJob\Art of MATLAB\instructorTest.m (instructorTest.fibonacciOf5) at 10
================================================================================
.
Done instructorTest
__________
Failure Summary:
Name Failed Incomplete Reason(s)
==========================================================================
instructorTest/fibonacciOf5 X Failed by verification.
results =
1x3 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Totals:
2 Passed, 1 Failed, 0 Incomplete.
0.028906 seconds testing time.
66.6667% Passed.
Benjamin, whose code we tested above, wrote a correct solution to the homework problem.
rmpath('eric'); addpath('benjamin'); results = run(instructorTest) percentPassed = 100 * nnz([results.Passed]) / numel(results); disp([num2str(percentPassed), '% Passed.']); rmpath('benjamin');
Running instructorTest
...
Done instructorTest
__________
results =
1x3 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Totals:
3 Passed, 0 Failed, 0 Incomplete.
0.015946 seconds testing time.
100% Passed.
Conclusion
In this post, we showed you the basics of using the new MATLAB unit testing infrastructure using homework grading as a use case.
We checked that the student's code worked (by returning the correct answer) for one valid value and worked (by throwing an error) for one invalid value. We also showed how you can use this infrastructure to provide an aid/check for the students that you can also use as part of your grading.
We hope this brief introduction to the unit testing framework has shown you how you can make use of this feature even if you don't consider yourself a software developer. Let us know in the comments for this post how you might use this new functionality. Or, if you've already tried using matlab.unittest, let us know about your experiences here.
Published with MATLAB® R2013a
- Category:
- How To,
- New Feature
