Sean‘s pick this week is to revisit Richard’s pick from last week and focus on the timing experiments.

Today, I’m going to look at timing MATLAB code using the Performance Testing Framework. Last week, Richard, had a few different experiments he wanted to run and time. Specifically, he also wanted to see how the first run compared to the Nth run.

The performance testing framework makes it very easy to do this. It leverages the same infrastructure as the unit test framework, but allows you to time parts of a test and run the tests either a fixed number of times or until the time of the Nth run converges.

Here I’ll take the algorithmic pieces of Richard’s code and put them into their own tests. There are two test classes which inherit from matlab.perftest.TestCase. The classes are both in a +Test package which allows me to automatically create a suite of all tests.

The framework makes it easy to parameterize tests so I can exercise both models under both run modes just by passing in the parameters.

Here are the two classes:

classdef WithWithoutPersistence < matlab.perftest.TestCase
% Test with and without persistent data under normal and accelerator

    properties (ClassSetupParameter)
        ModelName = {'withoutPersistentLoading', 'withPersistentLoading'}; % Both models
        RunMode = {'Normal', 'Accelerator'}; % Both modes
    end

    methods (TestClassSetup)
        % Run before testing starts
        function startSimulink(~)
            % Start Simulink to make sure initial time isn't included.
            start_simulink            
        end
        function loadModel(testCase, ModelName, RunMode)
            % Take RunMode to force it to close and reload between modes
            load_system(ModelName);
            testCase.addTeardown(@()close_system(ModelName));
        end        
    end
    
    % Test methods
    methods (Test)       
        function testSimulation(~, ModelName, RunMode)
            % Simulate
            sim(ModelName, 'SimulationMode', RunMode);
        end
    end
end

And:

classdef FastRestart < matlab.perftest.TestCase
% Test fast restart
    
    methods (TestClassSetup)
        function startSimulink(~)
            % Start Simulink to make sure initial time isn't included.
            start_simulink
            
            load_system('withoutPersistentLoading');
        end
    end
    
    methods (TestClassTeardown)
        function restoreModel(~)
            % Restore state
            set_param('withoutPersistentLoading', 'FastRestart', 'off');
            close_system('withoutPersistentLoading');
        end
    end
    
    methods (Test)        
        function testFastRestart(~)
            % Simulate
            sim('withoutPersistentLoading', 'FastRestart', 'on');
        end
    end
end

From there, we’ll create the test suite and create a fixed time experiment to run one warmup and 10 samples.

import matlab.unittest.TestSuite
import matlab.perftest.FixedTimeExperiment
suite = TestSuite.fromPackage('Test', 'IncludingSubpackages', true);
fte = FixedTimeExperiment.withFixedSampleSize(10, 'NumWarmups', 1);
results = run(fte, suite);
testactivity = vertcat(results.TestActivity);
disp(testactivity)

Running Test.FastRestart
..........
.
Done Test.FastRestart
__________

Running Test.WithWithoutPersistence
..........
..........
..........
..........
....
Done Test.WithWithoutPersistence
__________

                                                   Name                                                   Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform           Version                      TestResult                          RunIdentifier            
    __________________________________________________________________________________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    _____________________    ________________________________    ____________________________________
    Test.FastRestart/testFastRestart                                                                      true      false       false           3.6662       warmup      10-May-2018 19:50:01    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.022907       sample      10-May-2018 19:50:01    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.023864       sample      10-May-2018 19:50:01    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.034052       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.023421       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.035024       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.024277       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.021401       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.021795       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.021364       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.FastRestart/testFastRestart                                                                      true      false       false         0.024193       sample      10-May-2018 19:50:02    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false            4.271       warmup      10-May-2018 19:50:06    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.9064       sample      10-May-2018 19:50:10    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.8215       sample      10-May-2018 19:50:14    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.9881       sample      10-May-2018 19:50:18    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false            3.983       sample      10-May-2018 19:50:22    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.7034       sample      10-May-2018 19:50:26    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.8451       sample      10-May-2018 19:50:30    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.7176       sample      10-May-2018 19:50:33    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.7336       sample      10-May-2018 19:50:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.8346       sample      10-May-2018 19:50:41    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation         true      false       false           3.7633       sample      10-May-2018 19:50:45    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.9732       warmup      10-May-2018 19:50:49    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.8535       sample      10-May-2018 19:50:53    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.8244       sample      10-May-2018 19:50:57    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.7287       sample      10-May-2018 19:51:00    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.8755       sample      10-May-2018 19:51:04    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.7872       sample      10-May-2018 19:51:08    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.7933       sample      10-May-2018 19:51:12    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.8545       sample      10-May-2018 19:51:16    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.9182       sample      10-May-2018 19:51:20    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.7648       sample      10-May-2018 19:51:24    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation    true      false       false           3.7802       sample      10-May-2018 19:51:27    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false           3.6732       warmup      10-May-2018 19:51:31    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false          0.18088       sample      10-May-2018 19:51:31    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false          0.07304       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.065155       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.064901       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false          0.06622       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.064989       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.064424       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.064974       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.063272       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation            true      false       false         0.067388       sample      10-May-2018 19:51:32    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false           3.7391       warmup      10-May-2018 19:51:36    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.25133       sample      10-May-2018 19:51:36    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.11322       sample      10-May-2018 19:51:36    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.11479       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.14222       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.12787       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.12439       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.12707       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false           0.1336       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.11642       sample      10-May-2018 19:51:37    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation       true      false       false          0.12538       sample      10-May-2018 19:51:38    ah-sdewolsk     win64      9.4.0.813654 (R2018a)    [1x1 matlab.unittest.TestResult]    cc5b36fc-7c9f-4f09-9054-c60025f31196

Now we can either use varfun to analyze the groups or push this to Excel to be stored or pivoted out.

mt = varfun(@median, testactivity, ...
    'GroupingVariables', {'Name', 'Objective'}, ...
    'InputVariables', {'MeasuredTime'})

mt =
  10×4 table
                                                   Name                                                   Objective    GroupCount    median_MeasuredTime
    __________________________________________________________________________________________________    _________    __________    ___________________
    Test.FastRestart/testFastRestart                                                                       sample          10             0.023642      
    Test.FastRestart/testFastRestart                                                                       warmup           1               3.6662      
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation          sample          10                3.828      
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Normal]/testSimulation          warmup           1                4.271      
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation     sample          10               3.8088      
    Test.WithWithoutPersistence[ModelName=withoutPersistentLoading,RunMode=Accelerator]/testSimulation     warmup           1               3.9732      
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation             sample          10             0.065072      
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Normal]/testSimulation             warmup           1               3.6732      
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation        sample          10              0.12623      
    Test.WithWithoutPersistence[ModelName=withPersistentLoading,RunMode=Accelerator]/testSimulation        warmup           1               3.7391      

Or:

writetable(testactivity, 'TestActivity.xlsx')

Comments

Do you have a use for repetitively timing MATLAB code or Simulink models?

Give it a try and let us know what you think here.

Get the MATLAB code

Published with MATLAB® R2018a