I once faced a question from a friend who was perusing MATLAB documentation while implementing a test suite for some production software and found an example test which used the addTeardown method of the TestCase object. This person found themselves wondering why they might want to use it to manage their test fixture, citing that it was unclear where addTeardown fit with other fixture related features like TestMethodTeardown and TestClassTeardown. Alright, what is the deal with addTeardown and why would we use it? This construct is actually not seen in most xUnit paradigms so it may be unfamiliar to many. However, in fact addTeardown is more robust and, once you get used to it, actually less difficult than the other teardown routines. However its usage is markedly different from standard xUnit paradigms, so TestClassTeardown and TestMethodTeardown were included for standard familiarity.

Before we get too far into this, I am going to mysteriously add a few dummy folders to the path, save the entire path contents, and take a snapshot of these top three folders on the path using the following quick helper function. These will be handy as we explore the topic.

function showTopThreePathFolders
p = strsplit(path, pathsep);
disp(p(1:3)');
end

addpath(fullfile(pwd, 'folderOnOriginalPath1'));
addpath(fullfile(pwd, 'folderOnOriginalPath2'));
addpath(fullfile(pwd, 'folderOnOriginalPath3'));
blogPostPath = path;
showTopThreePathFolders;

    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath1'

Got it? Good, moving on.

A House of Order

The first reason that addTeardown is more robust is that it gives an order guarantee. The contract is that teardown actions occur in the reverse order to which they were added. When different fixtures change the same state, this prevents the teardown from executing out of order and restoring to the wrong end state. Here is an example demonstrating how this setup/teardown can run in the wrong order without this feature:

classdef BadFixturesTest < matlab.unittest.TestCase
    properties
        P1
        P2
    end
    
    methods(TestMethodSetup)
        function addPath1(testCase)
            disp('Adding path1');
            testCase.P1 = addpath(fullfile(pwd, 'path1'));
            showTopThreePathFolders;
        end
        
        function addPath2(testCase)
            disp('Adding path2');
            testCase.P2 = addpath(fullfile(pwd, 'path2'));
            showTopThreePathFolders;
        end
    end
    methods(TestMethodTeardown)
        function restoreP2(testCase)
            disp('Restoring path2');
            path(testCase.P2);
            showTopThreePathFolders;
        end
        
        function restoreP1(testCase)
            disp('Restoring path1');
            path(testCase.P1);
            showTopThreePathFolders;
        end 
    end
    
    methods(Test)
        function runTest(~)
        end
    end
end

A couple things to note about this and the other examples in this post:

Look at what can happen when the methods are executed. Again, the order is not guaranteed so the order shown is possible in general:

runtests BadFixturesTest;
showTopThreePathFolders;

Running BadFixturesTest
Adding path1
    '/Users/Shared/blog/addTeardown/path1'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'

Adding path2
    '/Users/Shared/blog/addTeardown/path2'
    '/Users/Shared/blog/addTeardown/path1'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'

Restoring path1
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath1'

Restoring path2
    '/Users/Shared/blog/addTeardown/path1'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'

.
Done BadFixturesTest
__________

    '/Users/Shared/blog/addTeardown/path1'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'

The above test run demonstrates the following:

As you can see, the path was clobbered because we had no guarantee of order. One thought may be to use rmpath instead of path to restore the state, but the problem with rmpath is that it might result in removing something from the path that was on the path at the start. Using the path function allows the test to ignore whether or not the folder was on the path to start. Instead, we can solve this using addTeardown (and the result is much smaller to boot):

classdef GoodFixturesTest < matlab.unittest.TestCase
    
    methods(TestMethodSetup)
        function addPath1(testCase)
            p = addpath(fullfile(pwd, 'path1'));
            testCase.addTeardown(@path, p);
        end
        
        function addPath2(testCase)
            p = addpath(fullfile(pwd, 'path2'));
            testCase.addTeardown(@path, p);
        end
    end
    
    methods(Test)
        function runTest(~)
        end
    end
end

path(blogPostPath); % Restore to the original path
runtests GoodFixturesTest;
showTopThreePathFolders;

Running GoodFixturesTest
.
Done GoodFixturesTest
__________

    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath1'

Now it doesn't matter which setup method is executed first because addTeardown executes in LIFO order. We have a guarantee that the teardown is executed in the reverse order of setup.

Let's Make Exceptions

OK, now another way to enforce order of setup/teardown is to write the code as a single setup and teardown function and code the order directly. This works but is not exception safe. We want to guarantee that the state we modify in the test is correctly restored to the original state even in the event of unexpected errors as well as other code flow interruptions like assertion and assumption failures. However this is problematic when there is a lot of fixture setup/teardown in single methods. Observe the following scenario, this time with a function-based test just to show that addTeardown can be used there as well:

function tests = NotExceptionSafeTest
tests = functiontests(localfunctions);
end

function setup(testCase)
testCase.TestData.P1 = addpath(fullfile(pwd, 'path1'));
testCase.TestData.P2 = addpath(fullfile(pwd, 'path2'));

ensureValidEnvironment(testCase);
testCase.TestData.WarnState = warning('off', 'MyProduct:MyFeature:MyWarnID');
end

function teardown(testCase)
warning(testCase.TestData.WarnState);
path(testCase.TestData.P2);
path(testCase.TestData.P1);
end

function testSomething(~)
end

OK, the happy path here is fine, the teardown is executed in the reverse order of the setup and things look good. However, if any of the code in the setup method errors, has an assertion, or filters the test using an assumption we have a problem. What in the world is in that ensureValidEnvironment helper function anyway?

function ensureValidEnvironment(testCase)
assumeFalse(testCase, ismac, ...
    'This feature is not supported on the Mac'); 

It looks as though this test should be filtered on the Mac (which, as it turns out, happens to be my platform of choice). We may be in trouble. Let's see what happens.

runtests NotExceptionSafeTest;

Running NotExceptionSafeTest

================================================================================
NotExceptionSafeTest/testSomething was filtered.
    Test Diagnostic: This feature is not supported on the Mac
================================================================================

================================================================================
Error occurred in NotExceptionSafeTest/testSomething and it did not run to completion.

    --------------
    Error Details:
    --------------
    Reference to non-existent field 'WarnState'.
    
    Error in NotExceptionSafeTest>teardown (line 14)
    warning(testCase.TestData.WarnState);
    
================================================================================
.
Done NotExceptionSafeTest
__________

Failure Summary:

     Name                                Failed  Incomplete  Reason(s)
    =================================================================================
     NotExceptionSafeTest/testSomething    X         X       Filtered by assumption.
                                                             Errored.
    

The test is filtered correctly, but we run into problems when we try to teardown our fixtures and the test fails. This happens because the WarnState field was never added to the TestData and therefore when the framework executes teardown the first line errors. Worse, because this line errors we never restore the path to the starting value.

showTopThreePathFolders;

    '/Users/Shared/blog/addTeardown/path2'
    '/Users/Shared/blog/addTeardown/path1'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'

The path1 and path2 folders were never removed.

Again, using addTeardown simplifies and solves this problem:

function tests = ExceptionSafeTest
tests = functiontests(localfunctions);
end

function setup(testCase)
p1 = addpath(fullfile(pwd, 'path1'));
addTeardown(testCase, @path, p1);

p2 = addpath(fullfile(pwd, 'path2'));
addTeardown(testCase, @path, p2);

ensureValidEnvironment(testCase);
warnState = warning('off', 'MyProduct:MyFeature:MyWarnID');
addTeardown(testCase, @warning, warnState);
end

function testSomething(~)
end

Now I like this for a few reasons:

That's all fine and dandy, but does it work?

path(blogPostPath); % Restore to the original path
runtests ExceptionSafeTest;

Running ExceptionSafeTest

================================================================================
ExceptionSafeTest/testSomething was filtered.
    Test Diagnostic: This feature is not supported on the Mac
================================================================================
.
Done ExceptionSafeTest
__________

Failure Summary:

     Name                             Failed  Incomplete  Reason(s)
    ==============================================================================
     ExceptionSafeTest/testSomething              X       Filtered by assumption.
    

The test no longer errors unexpectedly, and the path is restored correctly:

showTopThreePathFolders;

    '/Users/Shared/blog/addTeardown/folderOnOriginalPath3'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath2'
    '/Users/Shared/blog/addTeardown/folderOnOriginalPath1'

testCase.addTeardown(@blogPost);

So if you can't tell, I am a big advocate of addTeardown, and I use it almost exclusively in my own testing because I highly value robustness and am already familiar with the paradigm. Many people may not encounter these cases in their testing (or can forgive problems in failure scenarios). The TestMethodTeardown approach is more familiar from other xUnit implementations and is perfectly valid to use. That said, for the most robust, most production grade testing you should definitely consider addTeardown. What patterns and principles have you used to ensure safety and robustness of software fixtures? Let us know in the comments!

Get the MATLAB code

Published with MATLAB® R2015a