Steve on Image Processing

MTEST – A unit test harness for MATLAB code 23

Posted by Steve Eddins,

I just posted MTEST on the MATLAB Central File Exchange. MTEST is a unit test harness for testing MATLAB code. The design of MTEST is based on the popular xUnit architecture.

Now, writing test harnesses isn't exactly a central part of my day job, so let me describe how this project originated and evolved.

I first wrote my own test harness back in 2003 or so, when Digital Image Processing Using MATLAB was published. I needed an easy way to write and run tests for all the code in the book. The test harness we use at MathWorks has evolved over many years to support the needs of hundreds of developers and quality engineers, and dozens of products, so it was a little "big" for the needs of the book project. So I hacked together something simple that was just good enough to serve my own needs.

In 2006 I first presented my "Take Control of Your Code" talk at the International Conference on Image Processing. (I have since given versions of this talk at a half-dozen universities.) I described several tools and techniques intended to help "nonprogrammer" engineers and scientists be more productive with their coding. For testing code I suggested using a good test harness that makes writing and running tests as easy as possible. For testing MATLAB code, I recommended checking out the user-written test harnesses on the MATLAB Central File Exchange.

Then R2008a was released about a year ago, with dramatically improved language features for object-oriented programming. At about the same time I read the book Test-Driven Development, by Kent Beck. The book included a case study on using test-driven development to develop a test harness. And we were actively working on the second edition of Digital Image Processing Using MATLAB. Hey, it's a three-fer! A chance to learn the new language features, learn test-driven development techniques, and update the book's tests all at the same time.

Last spring I met Greg Wilson, co-editor of Beautiful Code, when he visited The MathWorks headquarters in Natick, MA. (Greg's articles on "software carpentry" partially inspired my "Take Control of Your Code" presentation.) Greg was interested in the experiences of engineers and scientists who do their programming in MATLAB, and he encouraged me to continue to develop and refine my MATLAB test harness. And he introduced me to Gerard Meszaros, author of xUnit Test Patterns. Both Greg and Gerard offered helpful architectural and design feedback (thanks, guys!), which I slowly worked to incorporate in the summer and fall. (Very slowly! This was a side project, remember.)

Finally last month I got to the point where I was willing to let people look at it and try it out. You can download it from the MATLAB Central File Exchange. Be sure to check out the extensive documentation. I just discovered that the File Exchange automatically made all of my html documentation files available online, without having to download and unpack the .zip file. That's very cool. The starting point for the documentation is the Readme.html file, and you might also want to look at "MTEST Quick Start - How to write and run tests."

I should point out that there are several other unit test harnesses available on the File Exchange. You can find them by searching for "unit test". I haven't looked at them in any detail, so I don't have particular recommendations. But if you don't like what I've done, take a look at the alternatives.

If you try MTEST, please let me know what you think. You can comment either here or on the File Exchange submission page.

23 CommentsOldest to Newest

Hi,

The documentation says that calling ‘mtest’ with no input arguments will only call functions that start with “test” or “Test”, however, this does not seem to be the case.

Inside of TestSuite.m, in fromPwd(), the comments say the same thing, but again, there seems to be no logic to test this functionality.

I just started working with MTEST, so it might be a case of end-user error, but any help is appreciated. So far, it seems like this will be extremely helpful library.

Thank you again,
Evan

Evan—I think you’re right. It looks like I broke this when I refactored the beta version to use the composite pattern for test suites. I’ll post a fix soon.

Thanks Steve, the update worked perfectly. For my own use, i created two new assert functions for comparing files and directories. Would love to share them with you and have them added to the distribution. -Evan

Dear Steve,
First of all, thank you very much for this post.
However, it seems that ‘mtest’ will not work with older Matlab version, specifically R2007b.

I wonder, is there a possibility of releasing a version compatible with this version?

Or perhaps you have some other useful advice?

Thanks in advance
Andrey

Rubshtein—MTEST makes heavy use of object-oriented language features introduced in R2008a. I have no plans to port MTEST to earlier MATLAB versions. However, there are other MATLAB test frameworks available. Search the File Exchange for “unit test”, or search Google for “matlab unit test”.

I am using a package structure to organize my MATLAB project and want to include unit-tests. I would like to have a single command to call all test cases inside the complete package hierarchy.

My stucture is something like package1.subpack1.mfile1 and I put test functions in package1.subpack1.test.test_mfile. I perform a search for “+test” directories by compiling a find command and calling system() (that is neither elegant nor portable), cd to the found directories, call TestSuite.fromPwd() to collect all test-cases and collect the test-suites from all directories in an array. At the end I simply call run() of the array members.
This does not work (no test-cases are found by TestSuite.fromPwd()), but does work, if I rename the test directories to something like “_test” (meaning they are no package hierarchy any more). Is this a bug?

Maybe there is a more elegant way to manage unit-tests in a package hierarchy?

Best regards,
Kay-Uwe

Kay-Uwe—You can’t CD into a package directory and then call M-files within that directory as if they weren’t part of a package. That’s why TestSuite.fromPwd() doesn’t find any test cases. It tries to call the M-files in the directory, but MATLAB can’t find them because they aren’t on the path, so no test cases are found.

To gather test cases from within a package, you would need to call the test M-files using the fully scoped package name, as in:

suite = package1.subpack1.test.test_mfile;
suite.run()

Maybe we need a TestSuite.fromPackage() method, and we need to teach runtests to accept package names.

Having TestSuite.fromPackage() would be nice to have, but so far using simple “test” subdirs (no further package level) works fine for me.

Thanks for the answer,
Kay-Uwe

Kay-Uwe—Thanks for following up. I am planning to make it easier to use test directories in a package. The MATLAB what function can be used to determine information about a package and its contents.

Dear Steve,
I am encountering the same package-structure problem. Are you planning to support it any time soon? Perhaps you are and my usage below is wrong. Thank you so much in advance.
Oren

===
I have a package directory “+core/+xunit” with a file:

classdef TestLearnXunit < TestCase
    %TESTLEARNXUNIT A learning test of the xUnit testing framework.
    %   Detailed explanation goes here
    
    properties (GetAccess = protected)
        x
    end
    
    %=========================== CONSTRUCTORS ============================
    methods
        function self = TestLearnXunit(name)
            %TestLearnXunit Constructor
            %   TestCaseInDir(name, testDirectory) constructs a test case
            %   using the specified name.
            self = self@TestCase(name);
        end
    end
    
    %=========================== SETUP METHODS ===========================
    methods
        function setUp(self)
            %setUp Simple test fixture setup.
            self.x = 1;
        end
        
        function tearDown(self)
            %tearDown Simple test fixture tear-down.
            self.x = 0;
        end
    end
    
    %=========================== TESTING METHODS =========================
    methods
        function testXValue(self)
            assertEqual(self.x, 1);
        end
    end
    
end

My path contains the parent directory of “+core”. When I “runtests”, it gets confused about the fully qualified test class name:

>> runtests core.xunit.TestLearnXunit
Starting test run with 2 test cases.
F.
FAILED in 0.002 seconds.

===== Test Case Failure =====
Location: c:\oren\core\src\test\+core\+xunit\TestLearnXunit.m
Name:     TestLearnXunit

C:\oren\core\matlab_xunit\xunit\TestCase.m at line 75
C:\oren\core\matlab_xunit\xunit\TestSuite.m at line 78
C:\oren\core\matlab_xunit\xunit\runtests.m at line 73

Undefined function or method 'TestLearnXunit' for input arguments of type 'core.xunit.TestLearnXunit'.

Quick update: mtest was trying to run the constructor of the class “TestLearnXunit” as a test case. I renamed the class to “LTestXunit” and it works now.

Following up on a package suite, is there a way to call TestSuite.fromPackage() that will scan the package and all its sub-packages for test classes and run them?

Thanks!

Oren—I’m interested in expanding MATLAB xUnit to support test suites in packages, but because of my current activities it’s going to be at least a couple of months before I can look at it.

Hi, I’m trying to get started using xUnit with MatLab. Having trouble getting it to recognize test files in some cases, but this is something I can work on figuring out. My main question is, would it be possible to post this on GitHub or somewhere like that?

Peter—Be sure to check out the doc for info on writing test cases to be automatically recognized.

I have no plans to post this package elsewhere. Is there some particular reason you’d like to see that?

Steve,

I want to write a test for a function that returns a cell array with mixed numbers & text. Which assert* test would you recommend using in this case?

The best I’ve come up is:

assertEqual(logical(1), isequal(expectedresult, actualresult))

Thanks.

OE—Yes, I haven’t written assertion functions for the container types such as cell or struct. I can suggest a slight simplification of what you wrote:

assertTrue(isequal(expectedresult, actualresult))

First of all thank you for this great tool.
I have a collection of classifier functions and I’d like to test them all for specific common properties.
Currently I have to write a test suite for each function. The suites essentially only differ in the name of the function under test. The tests in are all identical.
Is there a way to automate this? (i.e. provide the function name as an argument)

idimou—Yes, I can imagine a few ways to do this. Here’s one approach:

function test_suite = classifier_functions_test
initTestSuite

function function_list = setup
function_list = {@classifier1, @classifier2, @classifier3};

function testThis(function_list)
for k = 1:numel(function_list)
    % Test "This" for each classifier function.
end

function testThat
for k = 1:numel(function_list)
    % Test "That" for each classifier function.
end

These postings are the author's and don't necessarily represent the opinions of MathWorks.