Comments on: Creating List of Dates, Stepping by a Month https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/?s_tid=feedtopost Loren Shure is interested in the design of the MATLAB language. She is an application engineer and writes here about MATLAB programming and related topics. Wed, 27 Mar 2019 13:19:20 +0000 hourly 1 https://wordpress.org/?v=6.2.2 By: Peter Perkins https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47968 Wed, 27 Mar 2019 13:19:20 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47968 Rob, as I said, I hear your request for a better way to know what leap seconds a given version of MATLAB knows about, and for a way to update.

* I would think that even if spacecraft missions use UTC, there is still ample opportunity to accidentally create timestamps locally using in local time. datetime will PREVENT you from mixing “unzoned” timestamps with UTC timestamps, and will automatically account for timezone differences between local timestamps and UTC timestamps. datenum puts all of the burden on you. As long as you are careful …

* Bear in mind that because datenum counts days, most datenums involve roundoff, because, for example, 1/24 is not representable in floating point. As long as you’re careful …

* datetime(‘2015-12-31T23:59:60.000Z’,’TimeZone’,’UTCLeapSeconds’) errors because we know of no use case where trying to interpret a TEXT timestamp that is “out of range” is useful. On the other hand, doing the analogous thing with numbers IS very useful when creating sequences. This is an intentional design decision. Further, datetime(‘2015-12-31T23:59:60.000Z’,’TimeZone’,’UTCLeapSeconds’) errors because ALL of the inputs fails (all one of them) and datetime assumes you must have something seriously wrong, and returning an array of ALL all one) NaT is not very useful. As opposed to just having data that has occasional goofs, such as this

datetime([“2015-12-31T23:59:60.000Z” “2016-12-31T23:59:60.000Z”],’TimeZone’,’UTCLeapSeconds’)

which returns valid datetime values where it can.

* There’s no reason why your find_Leap_seconds_Matlab_knows can’t use numbers, not text, and work as you expect. This is an alternative implementation

>> d = datetime(repmat((2010:2018)’,1,2),repmat([6 12],9,1),1,’TimeZone’,’UTCLeapSeconds’);
>> dend = dateshift(d,’end’,’month’) + hours(24);
>> dend(dend.Second == 60)
ans =
3×1 datetime array
2012-06-30T23:59:60.000Z
2015-06-30T23:59:60.000Z
2016-12-31T23:59:60.000Z

* Yes, we are definitely aware of the possibility of negative leap seconds and will correctly support them if that actually happens. With any luck, leap seconds will be abolished before that need arises (I forget how long it is we’d need to be in the current “local” speeding up for it to become necessary but as I recall it’s a ways off).

]]>
By: Rob https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47966 Tue, 26 Mar 2019 20:26:44 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47966 I just installed the latest Matlab, 2019a for Mac… the above error still exists in Matlab, still not giving NaT when fed a bad time string, whereas a bad numbered time works.
But should that error be fixed, by code example would not work. Also, I forgot that they can add on 2 leap seconds if they wished, so I need to check for that. I suspect this code (it was nicely formatted on the cut/paste at least) would tell me the last leap second any given Matlab knows, but really want an inbuilt command that will do it for me – and also alert me to cases when there is a negative leap second.

function known_leap_seconds = find_Leap_seconds_Matlab_knows
% This DOES NOT WORK for any negative leap seconds
known_leap_seconds = [];

if ~exist(‘datetime’,’file’)
fprintf(‘This Matlab does not know of any leap seconds, and does not have the inbuilt datetime command.\n’);
end

% leap seconds occur at end or June or end of Dec,
% on 23:59:60, or both on 23:59:60 and 23:59:61
ending = {’06-30T23:59:60′,’06-30T23:59:61′,’12-31T23:59:60′,’12-31T23:59:61′};
for yyyy = 1970:(year(datetime)+1) % check this year and next for safety
for z=1:numel(ending)
try
in = sprintf(‘%04d-%s.000Z’,yyyy,ending{z});
out = datetime(in,’TimeZone’,’UTCLeapSeconds’);
if isequal(in,char(out)) % Don’t seem to need the char, but just in case…
known_leap_seconds = [known_leap_seconds;out];
% Yes this is a growing array, but only 27 so far…
end
catch % Don’t care about the catch
continue % Pointless continue
end
end
end
fprintf(‘This Matlab is aware of %d leap seconds since 1970.\n’,numel(known_leap_seconds));
fprintf(‘Last known leap second of this Matlab is %s.\n’,char(known_leap_seconds(end)));
return

]]>
By: Rob https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47964 Tue, 26 Mar 2019 19:52:10 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47964 Loren, thanks for investigating an answer for me!
Peter P, datenum worked well for me since for most spacecraft missions (especially particle instruments) , millisecond time resolution is as good as you can hope for. They may give time to more precision, but their spacecraft clocks drift and aren’t that accurate. The NAIF SPICE people have to deal with all this – and personally if I was coding up Matlab, I would take my leap seconds cues from how NAIF deals with them. Also, spacecraft missions tend to work in UTC time, don’t really care about time zones for spacecraft data (e.g. what timezone would you put the International Space Station in?)
I know of no other coding language that deals with leap seconds (although c, Fortran, IDL and Matlab all have packages from the NAIF SPICE people that can be used for time conversions – but then you’ve got to keep their packages and data files (what they call kernels) up to date – which is a whole other book keeping exercise). When I know the language doesn’t use leap seconds, I can code around that. When I know the language knows of all current leap seconds, I can code with that. My issue is with Matlab as currently is, I can’t tell if the Matlab that a user of mine is using has all the current leap seconds – and I have no easy and quick way to code around that.
So what I really want is:
A) a way to easily find out the last leap second that this current Matlab in use is aware of (i.e. an mfilename equivalent but for last leap second).
B) a way I can manually add new leap seconds, or replace the leap second table Matlab is aware of.
The latter is because if I’m making production Matlab code, we freeze the version of Matlab used, so it wouldn’t get updates to the leap second table from newer Matlab versions.

But generally – leap seconds in Matlab is still in beta as far as I’m concerned. Here’s an example why.
I know there is a leap second at the end of 2016, and so does Matlab – sometimes:
>> a = datetime(2016,12,31,23,59,60,’TimeZone’,’UTCLeapSeconds’)

a =

datetime

2016-12-31T23:59:60.000Z
Great : output time = input time, leap second is valid
Now let’s try this on a year without a leap send at the end of it – say 2015:
>> a = datetime(2015,12,31,23,59,60,’TimeZone’,’UTCLeapSeconds’)
a =
datetime
2016-01-01T00:00:00.000Z
Now output time is not the same as input time, it’s the first second of the next day – but this only works for numbers. What if my input is a string?
>> a = datetime(‘2015-12-31T23:59:60.000Z’,’TimeZone’,’UTCLeapSeconds’)
Error using datetime (line 604)
Unable to parse date/time text using the format ‘uuuu-MM-dd’T’HH:mm:ss.SSS’Z”.

>> a = datetime(‘2016-12-31T23:59:60.000Z’,’TimeZone’,’UTCLeapSeconds’)
a =
datetime
2016-12-31T23:59:60.000Z

So the first one behaves differently than the case when numbers are used as input. And it errors. Why not give a NaT?
(This is on Mac Matlab 2017b, I really should put the new 2019b on)

So from that I can very crudely (and inefficiently) check for all leap seconds my Matlab knows about by doing this code:
known_leap_seconds = [];
for yyyy = 1970:(year(datetime)+1)
try
a = datetime(sprintf(‘%04d-06-30T23:59:60.000Z’,yyyy),’TimeZone’,’UTCLeapSeconds’);
known_leap_seconds = [known_leap_seconds;a];
end
try
a = datetime(sprintf(‘%04d-12-31T23:59:60.000Z’,yyyy),’TimeZone’,’UTCLeapSeconds’);
known_leap_seconds = [known_leap_seconds;a];
end
end
known_leap_seconds

And this works – and for now is still up to date as the latest as we haven’t had a leap second for a few years.
However – this fails if we have a NEGATIVE leap second. Yes, that can exist – although has never been used.

How would I know if Matlab was accounting for the negative leap second?

Yep, can’o’worms… a valiant attempt by Matlab to deal with leap seconds – but not yet at a stage I can trust it.

Regards,

Rob

]]>
By: Paul Shoemaker https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47900 Fri, 08 Mar 2019 23:53:46 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47900 Rob, for your #2 question, try the below code and see if it gets you what you need. It’s inspired by a FEX submission by Erwin Mayer

import java.lang.String
import java.util.* java.awt.*
import java.util.Enumeration
myTime = java.util.GregorianCalendar();
get(myTime.getTimeZone());
]]>
By: Peter Perkins https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47898 Fri, 08 Mar 2019 22:45:54 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47898 Rob, I don’t know what to say to get you more comfortable with using datetime over datenum, other than to say that datetime addresses many of the shortcomings of datenum, notably no support for time zones, lower precision, and round-off when representing most points in time.

If leap seconds are important to you, I don’t understand how using datenum can suit your needs unless you have code that you have written that deals with them. We make sure every release is up to date with leap seconds as of its release date, so if you are on the current version, you should be ok. A sort of round-about way to confirm that is to subtract 1-Jan-1970 from the current date using the UTCLeapSeconds time zone. But I hear what you are saying about older versions and being able to get an explicit list. That’s something we are actively working on.

To create the sequence you want, do this:

   datetime(2018,1,50):hours(1):datetime(2018,1,122)

Fractional days have no meaning in most time zones. Perhaps you don’t care about time zones, but that’s why the datetime function will not accept fractional days.

]]>
By: Paul Shoemaker https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47896 Fri, 08 Mar 2019 19:21:23 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47896 Rob,

For your #2 question, there is a FEX submission by Erwin Mayer that makes use of Java instead of datetime that could be helpful: https://www.mathworks.com/matlabcentral/fileexchange/27953-convert-between-world-time-zones-with-daylight-saving-times

Using the above as a resource, take a look at the code snippet below and see if it gets you what you need:

import java.lang.String
import java.util.* java.awt.*
import java.util.Enumeration

myTime = java.util.GregorianCalendar();
get(myTime.getTimeZone())

Cheers,

Paul
MatlabInvesting.com

]]>
By: Loren Shure https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47890 Fri, 08 Mar 2019 05:50:24 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47890 @Gareth- Good to know this helped.

@Rob- I don’t know about leap seconds in detail. Will try to find you the answer.

–loren

]]>
By: Rob https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47888 Thu, 07 Mar 2019 17:48:36 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47888 I often work in day-of-year (how my source data appears) and often use a similar trick, i.e. day-of-year 50 is Feb 19th, but to put it in datenum or datetime: datenum(2018,1,50) or datetime(2018,1,50). I chose day-of-year 50 as it’s before any leap-day issues, just for clarity, but this works for leap years as well, i.e. datetime(year,1,DayOfYear)
As per your example, this also works with ranges of days, say I want every 6th day from day-of-year 50 to day-of-year 121.

datenum(2018,1,50:6:121)   or datetime(2018,1,50:121)

Great. Now we hit a difference. Say I want the same date range, but every hour. an hour is 1/24th of a day, so in datenum I can do datenum(2018,1,50:(1/24):121). This works. However, this is NOT allowed in datetime, since all input values must be integers (which is not the case with datenum), so the datetime solution would be: datetime(2018,1,1,(50*24):1:(121*24),0,0).
You can do similar tricks with minutes and seconds too.

Personally – I still use datenum for nearly everything time related. I have to deeply care about leap-seconds for my work, and although Matlab has had a valiant attempt at dealing with leap seconds, I don’t have an easy way to know how up to date it’s leap-second list is which ultimately makes it useless to me. Since I share code with others, I don’t know what version of Matlab they are using, so it’s difficult to code in a way to check how many leap seconds their version of Matlab is aware of. Mathworks – this needs to be addressed if you want use to trust Matlab time with leap second correction.

However, one big advantage for me of datetime is converting UTC strings to datetime (and then datenum).
My UTC strings are often of the form 2019-050T00:00:00:00.000 (day-of year format) or 2019-02-19T00:00:00.000 (year-month-day format).

Converting that to a datenum value was computationally expensive when I’ve thousands of them in a char array (n x 21 for the day-of-year former version). e.g.

%TIME = char array of n by 21 for UTC time in day of year (ddd)
TIME(:,[5 9 12 15])=' '; % Remove the - T : from yyyy-dddTHH:MM:SS.sss
TIME = str2num(TIME);
T = datenum(TIME(:,1)-1,12,31+TIME(:,2),TIME(:,3),TIME(:,4),TIME(:,5)); % note that the 6th argument (Seconds) can be decimal to allow for milliseconds.

However, with datetime it’s much quicker! MUCH MUCH quicker. Weirdly converting a UTC time string to a datenum was my biggest time-sink in my code. Anyway – my quick way was:

T = datenum(datetime(TIME,'Format','uuuu-DDD''T''HH:mm:ss.SSS'));
e.g.
T = datenum(datetime('2019-034T01:23:45.678','Format','uuuu-DDD''T''HH:mm:ss.SSS'));

If there’s a quicker way to do this I’d love to know it!

My two time questions for the room:
1) Is there an easy (1-line) way to find out the last leap-second that Matlab version datetime is aware of?
2) Is there a way to find out what time-zone the machine is set to without using datetime?

For the latter, I can use datetime to figure this out, but if someone I’ve shared my code with has an older version of Matlab that does not have datetime yet (and there are a few who can’t afford to upgrade), is there anyway I can figure out their timezone from Matlab using datenum or now or such?

]]>
By: Gareth Thomas https://blogs.mathworks.com/loren/2019/03/06/creating-list-of-dates-stepping-by-a-month/#comment-47886 Thu, 07 Mar 2019 08:40:40 +0000 https://blogs.mathworks.com/loren/?p=3242#comment-47886 Great post Loren, it is wonderful to see the flexibility and the power of datetime and the other functions around it. Personally I tend to use the mydates5 = datetime(2017,7,1) + calmonths(0:11). The eomday function was new to me and I had to look it up to see what the eomday stands for… End Of Month DAY. Knowing that makes it easier to remember, at least for me:)

]]>