{"id":249,"date":"2015-09-08T21:41:17","date_gmt":"2015-09-08T21:41:17","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=249"},"modified":"2015-09-09T14:19:17","modified_gmt":"2015-09-09T14:19:17","slug":"test-tags","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2015\/09\/08\/test-tags\/","title":{"rendered":"Tag, you’re it!"},"content":{"rendered":"\r\n\r\n

I have found that with any reasonably sized software project, sooner or later organization becomes important. As additional features and capabilities are introduced into a toolbox it becomes important to quickly and easily get your hands on the right piece of code for the task at hand. When we are talking about these software products, often times a piece of code does \"double duty\" so to speak. For these it is not uncommon that a single class or function or method is generalized to correctly serve many use cases.<\/p>

How do we narrow down on these snippets of code? The tests of course! While a single bit of source code may be general enough to fulfill many use cases, the tests should be written to ensure each use case and user visible feature works correctly. In other words, if you want each use case to be covered, it had better have its corresponding tests, even if the implementation is covered via some generalized mechanism in the source code.<\/p>

OK, easy enough. We'll use the tests and it will always be trivial to locate the specific tests we need right? Ummmmmm, no. Tests themselves are source code and will have the same issues with transparency as the source code they apply to. Because of this it is fairly common to see the tests themselves organized into some structure that attempts to categorize them for easy retrieval. Do I want to run the unit tests for the numeric foundations of my product? Just go to the tests\/numerics\/unit<\/tt> folder and type runtests<\/tt><\/b>. How about the tests that exercise these numerics through my user interface? Perhaps these are placed in the tests.system.ui<\/tt> test package? OK, maybe there's a lot of system testing debris in there that don't have to do with numerics but whatevs, deal with it!<\/p>

What if I want to find the tests\/code that covers the integration of my classification features and my persistence? For example lets say you perform a classification and save the results to some form of persistence such as a database or mat file. Well, hmm how do we find those tests? Will they be in tests\/classification<\/tt> or tests\/persistence<\/tt>? If they are in one folder but not the other does this mean that one of these test folders is the true \"owner\" of the feature and the other is not, relegated to a life of underachievement and low self-esteem?<\/p>

Not so! I am here to defend these poor test folders! I am here to declare upon the rooftops that if you didn't categorize your system perfectly not all hope is lost. I understand that categorizing things correctly is very hard to do<\/a>.<\/p>

Can we please de-emphasize hierarchies based on folder or package structure? Instead wouldn't it be great if we could tag specific test content with meta data for use in categorization? This meta data could then apply to test content across different axes, features, or viewpoints, unlike folders or package structures. You can't place a file in two folders at the same time. Instead lets spend most of our organizational time on figuring out the best categories for our test code.<\/p>

If you have R2015a or later this can be done easily using the test tagging features<\/a> of the test framework.<\/p>

This test tagging feature is a bit like labels in Gmail™, wherein you can apply certain labels to incoming mail messages and subsequently filter your messages using this label.<\/p>

How might we apply this in our test content?<\/p>

Well first let's look at a couple different axes that would be sure to get us into trouble trying to design inside a hierarchy. If we just stick with the examples already discussed, perhaps we might have some test classes that look like those below. Note these tests contain a bit of pseudocode and don't run since the product we are testing doesn't really exist.<\/p>

\r\nclassdef<\/span> (TestTags = {'Classification'<\/span>, 'Persistence'<\/span>}) ClassificationWithPersistenceTest < matlab.unittest.TestCase\r\n    % This test exercises how two features of my toolbox work together, namely<\/span>\r\n    % how the toolbox classification features work with persistence handling.<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testClassificationResultSavedCorrectly(testCase)\r\n            \r\n            % Classify some data and persist the result<\/span>\r\n            data = load('testData.mat'<\/span>);\r\n            classification = classify(data);\r\n            classification.persist('testFile.mat'<\/span>); \r\n            \r\n            result = load('testFile.mat'<\/span>);\r\n            \r\n            testCase.verifyEqual(result, classification, ...<\/span>\r\n                'The classification did not persist correctly in the mat file.'<\/span>);\r\n\r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>  \r\n\r\n<\/pre>
Note this test class has an additional TestTags<\/tt><\/b> attribute on the classdef block. In that attribute you can list some tags that will help identify the intended purpose(s) of the test. In this case we can identify this test as related to both the 'Classification'<\/tt> and 'Persistence'<\/tt> features.<\/p>
This is fun, lets do some more:<\/p>
\r\nclassdef<\/span> (TestTags = {'Numerics'<\/span>}) CoreNumericsUnitTest < matlab.unittest.TestCase\r\n    % Here is a unit test of the core numerics of my toolbox<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testCoreNumerics(testCase)\r\n            input = 1:10;\r\n            actual = CoreNumericEngine.analyze(input);\r\n            expected = expectedResultFromFirstPrinciples(input);\r\n            \r\n            testCase.verifyEqual(actual, expected, 'RelTol'<\/span>, sqrt(eps));\r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>  \r\n\r\n<\/pre>
Now we have a nice beautiful numerics unit test. How about that system test we talked about?<\/p>
\r\nclassdef<\/span> (TestTags = {'UI'<\/span>}) AcceptanceSystemTest < matlab.unittest.TestCase\r\n    % A system test exercising the full stack of my app with a focus on<\/span>\r\n    % priority customer workflows.<\/span>\r\n    \r\n    methods<\/span>(Test, TestTags = {'Persistence'<\/span>})\r\n        function<\/span> testApplicationImportData(testCase)\r\n            fig = numericsapp;\r\n            clickImportButtonAndSelect('testFile.mat'<\/span>);\r\n            \r\n            classification = getClassificationfromUI(fig);\r\n            expected = load('testFile.mat'<\/span>);\r\n            testCase.verifyEqual(classification, expected, ...<\/span>\r\n                'Classification data not imported correctly into the app.'<\/span>);\r\n        end<\/span>\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test, TestTags = {'Numerics'<\/span>})\r\n        function<\/span> testApplicationNumericResults(testCase)\r\n            fig = numericsapp;\r\n            clickAnalyzeButton(fig);\r\n            \r\n            actual = getCurrentResult(fig);\r\n            expected = CoreNumericEngine.analyze(getCurrentData(fig));\r\n            testCase.verifyEqual(actual, expected, ...<\/span>\r\n                'App results do not match numeric counterpart.'<\/span>);\r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n<\/pre>
This test was a bit different, as you can see the TestTags<\/tt><\/b> attribute is not only on the classdef block, but is included on the test methods block as well. Well, ultimately these tags apply to test methods, but including TestTags<\/tt><\/b> on the classdef block is a sort of shorthand for applying the tags to all of the methods of that class (and all subclasses if any). The first two test examples clearly wanted these tags applied to all methods of the class. However, system testing, in particular is a bit more nuanced. System tests can span the full feature set of a toolbox, and it is not uncommon for different tests to touch different portions of the toolbox functionality. In this case, both test methods are 'UI'<\/tt> but only one of them deals with 'Persistence'<\/tt> and one with 'Numerics'<\/tt>. Applying the tags at the method level allows us to differentiate these two.<\/p>
Well alright, what does this give us? Test selection anyone? Let's look at our full suite containing all these tests:<\/p>
import matlab.unittest.TestSuite<\/span>;\r\nfullSuite = TestSuite.fromFolder('tests'<\/span>)\r\n<\/pre>
\r\nfullSuite = \r\n\r\n  1x4 Test array with properties:\r\n\r\n    Name\r\n    Parameterization\r\n    SharedTestFixtures\r\n    Tags\r\n\r\nTests Include:\r\n   0 Parameterizations, 0 Shared Test Fixture Classes, 4 Unique Tags.\r\n\r\n<\/pre>
The careful reader will note that this suite's display actually shows 4 Unique Tags<\/tt>. If you are running in MATLAB this is actually a hyperlink that shows you the full list of tags contained in the suite. When clicked it looks something like:<\/p>
         Tag\r\n   ________________\r\n   'Classification'\r\n   'Numerics'\r\n   'Persistence'\r\n   'UI'<\/pre>
Now here's a quick way to see which tests we are working with in a suite. It is a concise way to place the names of each suite element into a column cell array which displays nicely.<\/p>
{fullSuite.Name}.'\r\n<\/pre>
\r\nans = \r\n\r\n    'AcceptanceSystemTest\/testApplicationNumericResults'\r\n    'AcceptanceSystemTest\/testApplicationImportData'\r\n    'ClassificationWithPersistenceTest\/testClassificationResultSavedCorrectly'\r\n    'CoreNumericsUnitTest\/testCoreNumerics'\r\n\r\n<\/pre>
Now we can slice and dice this suite however we want based on our tags using the nice selectIf<\/a> method:<\/p>
persistenceSuite = fullSuite.selectIf('Tag'<\/span>,'Persistence'<\/span>);\r\n{persistenceSuite.Name}.'\r\n<\/pre>
\r\nans = \r\n\r\n    'AcceptanceSystemTest\/testApplicationImportData'\r\n    'ClassificationWithPersistenceTest\/testClassificationResultSavedCorrectly'\r\n\r\n<\/pre>
It even supports wildcards:<\/p>
numericsSuite = fullSuite.selectIf('Tag'<\/span>,'Num*'<\/span>);\r\n{numericsSuite.Name}.'\r\n<\/pre>
\r\nans = \r\n\r\n    'AcceptanceSystemTest\/testApplicationNumericResults'\r\n    'CoreNumericsUnitTest\/testCoreNumerics'\r\n\r\n<\/pre>
Finally, one of the huge benefits of tagging tests like this is that you don't need to  commit to only one axis to categorize your test content. This example has shown adding specific tags according to features of the toolbox. However, we could also take the type of test as another axis we may want to work with. For example, if you are a fan of Google's small\/medium\/large<\/a> test nomenclature you may want to set up a CI system to run small tests for every check-in, medium tests every night, and large tests at the end of each iteration. This deferred testing strategy becomes very easy with the aid of test tags. Let's add 'Small'<\/tt>, 'Medium'<\/tt>, and 'Large'<\/tt> tags.<\/p>
\r\nclassdef<\/span> (TestTags = {'Classification'<\/span>, 'Persistence'<\/span>, 'Medium'<\/span>}) ClassificationWithPersistenceTest < matlab.unittest.TestCase\r\n    % This test exercises how two features of my toolbox work together, namely<\/span>\r\n    % how the toolbox classification features work with persistence handling.<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testClassificationResultSavedCorrectly(testCase)\r\n            \r\n            % Classify some data and persist the result<\/span>\r\n            data = load('testData.mat'<\/span>);\r\n            classification = classify(data);\r\n            classification.persist('testFile.mat'<\/span>); \r\n            \r\n            result = load('testFile.mat'<\/span>);\r\n            \r\n            testCase.verifyEqual(result, classification, ...<\/span>\r\n                'The classification did not persist correctly in the mat file.'<\/span>);\r\n\r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n\r\n\r\nclassdef<\/span> (TestTags = {'Numerics'<\/span>, 'Small'<\/span>}) CoreNumericsUnitTest < matlab.unittest.TestCase\r\n    % Here is a unit test of the core numerics of my toolbox<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testCoreNumerics(testCase)\r\n            input = 1:10;\r\n            actual = CoreNumericEngine.analyze(input);\r\n            expected = expectedResultFromFirstPrinciples(input);\r\n            \r\n            testCase.verifyEqual(actual, expected, 'RelTol'<\/span>, sqrt(eps));\r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n\r\n\r\nclassdef<\/span> (TestTags = {'UI'<\/span>,'Large'<\/span>}) AcceptanceSystemTest < matlab.unittest.TestCase\r\n    % A system test exercising the full stack of my app with a focus on<\/span>\r\n    % priority customer workflows.<\/span>\r\n    \r\n    methods<\/span>(Test, TestTags = {'Persistence'<\/span>})\r\n        function<\/span> testApplicationImportData(testCase)\r\n            fig = numericsapp;\r\n            clickImportButtonAndSelect('testFile.mat'<\/span>);\r\n            \r\n            classification = getClassificationfromUI(fig);\r\n            expected = load('testFile.mat'<\/span>);\r\n            testCase.verifyEqual(classification, expected, ...<\/span>\r\n                'Classification data not imported correctly into the app.'<\/span>);\r\n        end<\/span>\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test, TestTags = {'Numerics'<\/span>})\r\n        function<\/span> testApplicationNumericResults(testCase)\r\n            fig = numericsapp;\r\n            clickAnalyzeButton(fig);\r\n            \r\n            actual = getCurrentResult(fig);\r\n            expected = CoreNumericEngine.analyze(getCurrentData(fig));\r\n            testCase.verifyEqual(actual, expected, ...<\/span>\r\n                'App results do not match numeric counterpart.'<\/span>);\r\n        end<\/span>\r\n    end<\/span>   \r\nend<\/span>\r\n\r\n<\/pre>
We can even prefilter these tests in test suite creation methods like fromFolder:<\/p>
smallTests = TestSuite.fromFolder('tests\/v2'<\/span>, 'Tag'<\/span>, 'Small'<\/span>);\r\n{smallTests.Name}.'\r\n\r\nmediumTests = TestSuite.fromFolder('tests\/v2'<\/span>, 'Tag'<\/span>, 'Medium'<\/span>);\r\n{mediumTests.Name}.'\r\n\r\nlargeTests = TestSuite.fromFolder('tests\/v2'<\/span>, 'Tag'<\/span>, 'Large'<\/span>);\r\n{largeTests.Name}.'\r\n<\/pre>
\r\nans = \r\n\r\n    'CoreNumericsUnitTest\/testCoreNumerics'\r\n\r\n\r\nans = \r\n\r\n    'ClassificationWithPersistenceTest\/testClassificationResultSavedCorrectly'\r\n\r\n\r\nans = \r\n\r\n    'AcceptanceSystemTest\/testApplicationNumericResults'\r\n    'AcceptanceSystemTest\/testApplicationImportData'\r\n\r\n<\/pre>
A key point here is that while this test suite is very small and manageable, this definitely does not hold in any serious software development project. Using test tags is a nice way to avoid relying too much on limited folder or package structure for test organization, instead favoring tagging each test along whichever axes it belongs to.<\/p>
Then when you are looking for exactly the right set of tests to exercise exactly the right pieces of your software product......(wait for it)....<\/p>
\"\" <\/p>