Dependency-based Test Selection
This is the final part in my series on MATLAB Test where we’ll look at how to select tests based on files they depend on. Previously, I have covered:
- An introduction to MATLAB Test and the Test Manager
- The use of advanced coverage metrics
- Measuring and monitoring code quality
- Equivalence testing.
In large and complex projects, running the entire test suite every time something changes can be prohibitively expensive in time and/or compute resource. In these instances, just running the tests that are affected by the changes may be preferable. MATLAB Test provides mechanisms for easily doing this:
- runtests(__,DependsOn=val)
- testsuite(__,DependsOn=val)
- matlab.unittest.TestSuite.fromX(__,DependsOn=val) where X can be Class, File, Folder, Method, Name, Package, Project, or Requirements.
- selectIf(suite,DependsOn=Val) where suite is a matlab.unittest.TestSuite array.
In the above, val is a string array of files and folders that the tests must depend on to be selected. A MATLAB Test licence is required to use this feature.
In the following example, I have a project with source code and tests. The Dependency Analyzer shows me that some tests depend on AccountManager whilst those in DocPolynomTest don’t.
Example project and dependencies.
If I create a test suite from the project, I get 8 test points from both test classes:
>> s = testsuite(pwd);
>> {s.Name}'
ans =
8×1 cell array
{'BankAccountTest/testConstructor' }
{'BankAccountTest/testConstructorNotEnoughInputs'}
{'BankAccountTest/testDeposit' }
{'BankAccountTest/testWithdraw' }
{'BankAccountTest/testNotifyInsufficientFunds' }
{'DocPolynomTest/testConstructor' }
{'DocPolynomTest/testAddition' }
{'DocPolynomTest/testMultiplication' }
If I instead specify that I only want tests that depend on AccountManager, I get the expected subset:
>> s = testsuite(pwd,DependsOn="source/AccountManager.m");
>> {s.Name}'
ans =
5×1 cell array
{'BankAccountTest/testConstructor' }
{'BankAccountTest/testConstructorNotEnoughInputs'}
{'BankAccountTest/testDeposit' }
{'BankAccountTest/testWithdraw' }
{'BankAccountTest/testNotifyInsufficientFunds' }
For more control, you can use the matlabtest.selectors.DependsOn selector which allows you to control whether or not subfolders of the specified folders are included in the search, and what the maximum search depth is. The respective defaults are to not include subfolders and to search the full dependency hierarchy.
The DependsOn selector can also be used in conjunction with the Test Manager’s custom test suite functionality. From the drop-down menu, select “Manage Custom Test Suites”:
Manage Custom Test Suites.
Create a new test suite that uses the DependsOn selector:
Create a new test suite using the DependsOn selector.
Save and close, then select the new test suite from the Test Manager drop-down menu to view it:
New test suite in the Test Manager.
Note that for complex projects, performing the dependency analysis may take some time. There’s therefore a trade-off to be found between the time it takes to do the analysis and the time it takes to run the entire suite.
So how do we get the list of changed files to pass into the DependsOn selector? If you’re using Projects and you have uncommitted changes, the Projects API will tell you:
>> prj = currentProject();
>> prj.listModifiedFiles
ans =
ProjectFile with properties:
Path: "C:\blogs-test-selection\source\BankAccount.m"
Revision: "8974b348f16d445c01409bbcd00f7b9777aa40fd"
SourceControlStatus: Modified
Labels: [1×1 matlab.project.Label]
However, if you have committed changes and/or want to compare one branch to other (e.g. for a pull request), you’ll have to do some manual work with Git to get the list of changed files.
The DependsOn selector determines dependencies with static analysis. Static analysis has some limitations (documented here) which means that it's not infallible. Therefore, I still recommend running a full suite of tests at key stages of your development process.
In conclusion, MATLAB Test provides a mechanism for selecting tests that are impacted by source code changes, allowing you to run the subset of tests that matter and saving testing time.
This completes the series on MATLAB Test. I’ll be back with updates when new MATLAB releases come out!
- カテゴリ:
- MATLAB Test,
- Testing
コメント
コメントを残すには、ここ をクリックして MathWorks アカウントにサインインするか新しい MathWorks アカウントを作成します。