Should we ban tic/toc?

著者 Mike Croucher, May 7, 2024

35 ビュー (過去 30 日間) | 0 いいね | 28 コメント

Yesterday morning, I overheard my kids talking about a tic/toc ban which surprised me because they are not MATLAB users and not particularly well connected at MathWorks. How would they know about something as big as this before me?

The idea of tic/toc is simple. tic starts a timer while toc stops it and reports the result. I use it all the time, both in day to day work and when discussing performance related issues.  Here's an example for timing the generation of a random matrix

tic % start the timer

data = rand(2000);

toc % stops the timer and reports the result

Elapsed time is 0.025606 seconds.

It's great and I love it! However, there are issues with using it for performance work. There's usually variation in the time of an individual run. The first time you run any command, for example, is often slower than all subsequent times for various reasons. There's also variation because of all the other programs your computer is running or maybe the CPU is switching to different clock speeds as it hits some internal thermal limit and so on.

For this reason, it is usually much more beneficial to run the code many times and look at the statistics. For this reason, the timeit command was created.  In order to perform a robust measurement, timeit calls the specified function multiple times and returns the median of the measurements. Here's how its used

f = @() rand(2000);

timeit(f)

ans = 0.0219

Run that several times and you'll get much less variation which is what you'd expect given that its more robust.

Some of my fellow MathWorkers love timeit and I'm frequently advised to use it instead of tic/toc in blog posts. Loren Shure blogged about it back in 2013, just after it was introduced, and she recently messaged me after I'd published something, essentially saying 'Great post kid, but you should ditch tic/toc and use timeit".  You'll notice that my recent post about SUNDIALS integration in MATLAB uses timeit whenever I talk about performance. I'm trying to be good, honest I am! 

But banning tic/toc? That's just too much! I reached out to MATLAB Head of Product, Michelle Hirsch, and asked what on earth was going on because here's the thing....

I don't like MATLAB's timeit

I understand that timeit is more robust but I just don't like it and here's why:

I don't feel like I get more for my effort. tic/toc gives me a number. timeit gives me a number. It may be 'more robust' but I have no sense of how much more robust it is. How many times was it run and why can't I control it? What was the maximum run time and what was the minimum? In short, I want some evidence for the multiple runs that the documentation tells me have taken place and I want to be able to control the number of runs.
tic/toc is easy to use, even in the middle of 10,000 lines of janky code, I can just insert it wherever I feel like I need it. timeit requires me to wrap things into a function. This is fine for simple cases but more hassle than I can be bothered with when deep in the trenches.

So, I'll use timeit when I have to but please, don't take tic/toc away from me! 

We are not banning tic/toc

Turns out that my kids were referring to some other thing, totally unrelated to MATLAB! Something to do with video sharing. Dancing cats and so on. Michelle assured me that tic/toc is not going anywhere and put me in touch with the area of development who look after timeit to discuss ideas for improvement in the future. I've already fed back my 2 cents worth but am curious what you'd like to see in a timeit improvement/replacement?

Michelle also reminded me that for more serious performance related work, MATLAB has a Performance Testing Framework which I'll take a closer look at in the meantime.