{"id":1676,"date":"2016-06-13T08:57:39","date_gmt":"2016-06-13T13:57:39","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=1676"},"modified":"2016-06-09T14:16:29","modified_gmt":"2016-06-09T19:16:29","slug":"setting-a-new-time-basis-for-date-time-data","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2016\/06\/13\/setting-a-new-time-basis-for-date-time-data\/","title":{"rendered":"Setting a New Time Basis for Date \/ Time Data"},"content":{"rendered":"\r\n<div class=\"content\"><!--introduction--><p>I was recently working with someone who had some temporal data in various formats and was trying to merge them in a meaningful way.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#0f6e81ea-7767-47ab-bc68-84f2c52ad123\">Problem Setup<\/a><\/li><li><a href=\"#1160aa7d-7888-4d6d-928c-796d3fd3cb66\">What Didn't Work<\/a><\/li><li><a href=\"#3cabc59d-594f-42e4-89e1-4141570dfeef\">First Working Shot<\/a><\/li><li><a href=\"#446ee93a-859b-41d6-a048-c4c300d28f3a\">Much Better Answer<\/a><\/li><li><a href=\"#7cb6d3c7-04c2-4708-8d7b-0e043ffc278a\">P.S.<\/a><\/li><li><a href=\"#929598e9-6233-4962-8ded-889159f5b047\">And You?<\/a><\/li><\/ul><\/div><h4>Problem Setup<a name=\"0f6e81ea-7767-47ab-bc68-84f2c52ad123\"><\/a><\/h4><p>I made a small subset of the data and loaded it in.<\/p><pre class=\"codeinput\">load <span class=\"string\">setTimeBasis<\/span>\r\nwhos\r\n<\/pre><pre class=\"codeoutput\">  Name           Size            Bytes  Class       Attributes\r\n\r\n  mydatestr      1x6                12  char                  \r\n  t              5x1               161  datetime              \r\n\r\n<\/pre><p>First I looked at the both variables.<\/p><pre class=\"codeinput\">mydatestr\r\n<\/pre><pre class=\"codeoutput\">mydatestr =\r\n090506\r\n<\/pre><pre class=\"codeinput\">t\r\n<\/pre><pre class=\"codeoutput\">t = \r\n   12:32:28\r\n   12:59:50\r\n   12:59:51\r\n   12:59:52\r\n   12:59:53\r\n<\/pre><p>You can see that I have a string representing a date (May 6, 2009), and then a <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/datetime.html\"><tt>datetime<\/tt><\/a> array in <tt>t<\/tt>.  Though I saw that <tt>t<\/tt> was a <tt>datetime<\/tt>, I kept thinking of it as a duration since I didn't see a month, year, etc.  And I have to confess, though I usually do check the documentation, I was really resisting it yesterday - not sure why - perhaps I was being more obstinate than usual :-) !<\/p><h4>What Didn't Work<a name=\"1160aa7d-7888-4d6d-928c-796d3fd3cb66\"><\/a><\/h4><p>Next I converted the base date to <tt>datetime<\/tt>, and tried adding it to <tt>t<\/tt>.<\/p><pre class=\"codeinput\">d = datetime(<span class=\"string\">'090506'<\/span>,<span class=\"string\">'InputFormat'<\/span>,<span class=\"string\">'MMddyy'<\/span>)\r\n<span class=\"keyword\">try<\/span>\r\n     d + t\r\n<span class=\"keyword\">catch<\/span> E\r\n    disp(E.message)\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><pre class=\"codeoutput\">d = \r\n   05-Sep-2006\r\nAddition is not defined between datetime arrays.\r\n<\/pre><h4>First Working Shot<a name=\"3cabc59d-594f-42e4-89e1-4141570dfeef\"><\/a><\/h4><p>Ok, so that didn't work.  I finally realized that even though <tt>t<\/tt> was only displaying hours, minutes, and seconds, because it was a <tt>datetime<\/tt> array and not a <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/duration.html\"><tt>duration<\/tt><\/a>, I had to deal with that!  I also realized that I used the wrong formatting for converting the original date for <tt>mydatestr<\/tt> to a <tt>datetime<\/tt>!  What was I thinking!<\/p><pre class=\"codeinput\">newdate = datetime(mydatestr,<span class=\"string\">'InputFormat'<\/span>,<span class=\"string\">'yyMMdd'<\/span>)\r\n<\/pre><pre class=\"codeoutput\">newdate = \r\n   06-May-2009\r\n<\/pre><p>Let's inspect the first time point now.<\/p><pre class=\"codeinput\">t1 = t(1);\r\n[year(t1) month(t1) day(t1)]\r\n<\/pre><pre class=\"codeoutput\">ans =\r\n        2016           6           7\r\n<\/pre><p>Aha!  Now I need to convert the date (not time) values in <tt>t<\/tt> to that in <tt>olddate<\/tt>!  I can do this by computing the difference in dates, leading to a <tt>duration<\/tt> array which I <b>can<\/b> add to the base date.<\/p><pre class=\"codeinput\">olddate = datetime(year(t1),month(t1),day(t1));\r\ndatediff = newdate - olddate\r\nnewt = t + datediff\r\n<\/pre><pre class=\"codeoutput\">datediff = \r\n   -62136:00:00\r\nnewt = \r\n   12:32:28\r\n   12:59:50\r\n   12:59:51\r\n   12:59:52\r\n   12:59:53\r\n<\/pre><p>Let's check out what's in <tt>newt<\/tt> now (I know, <tt>eye(newt)<\/tt>!)<\/p><pre class=\"codeinput\">newt1 = newt(1);\r\nnewt1.Format = <span class=\"string\">'yy-MM-dd hh:mm:ss'<\/span>\r\n<\/pre><pre class=\"codeoutput\">newt1 = \r\n   09-05-06 12:32:28\r\n<\/pre><h4>Much Better Answer<a name=\"446ee93a-859b-41d6-a048-c4c300d28f3a\"><\/a><\/h4><p>Frankly, I got it to work but it was really ugly and somewhat contorted. Had I responsibly read the documentation before, I just might have found a better way.  Instead of showing you the doc right now, I will show you another way to poke for more information.  Let's find out what I can do with a <tt>datetime<\/tt> array.<\/p><pre class=\"codeinput\">methods(t)\r\n<\/pre><pre class=\"codeoutput\">\r\nMethods for class datetime:\r\n\r\nbetween            hour               le                 reshape            \r\ncaldiff            interp1            length             second             \r\ncat                intersect          linspace           setdiff            \r\ncellstr            isbetween          lt                 setxor             \r\nchar               iscolumn           max                size               \r\ncolon              isdst              mean               sort               \r\nctranspose         isempty            median             sortrows           \r\ndatenum            isequal            min                std                \r\ndateshift          isequaln           minus              timeofday          \r\ndatestr            isfinite           minute             transpose          \r\ndatetime           isinf              mode               tzoffset           \r\ndatevec            ismatrix           month              union              \r\nday                ismember           ndims              unique             \r\ndiff               isnat              ne                 vertcat            \r\neq                 isrow              numel              week               \r\nexceltime          isscalar           permute            year               \r\nge                 issorted           plot               ymd                \r\ngt                 isvector           plus               yyyymmdd           \r\nhms                isweekend          posixtime          \r\nhorzcat            juliandate         quarter            \r\n\r\nStatic methods:\r\n\r\nsetDefaultFormats  \r\n\r\n<\/pre><p>Hmmmm!  <tt>timeofday<\/tt> sure sounds interesting, doesn't it!?!  So let's try again.<\/p><pre class=\"codeinput\">newdate = datetime(mydatestr,<span class=\"string\">'InputFormat'<\/span>,<span class=\"string\">'yyMMdd'<\/span>)\r\ntod = timeofday(t)\r\nnewt2 = newdate + tod\r\nwhos\r\n<\/pre><pre class=\"codeoutput\">newdate = \r\n   06-May-2009\r\ntod = \r\n   12:32:28\r\n   12:59:50\r\n   12:59:51\r\n   12:59:52\r\n   12:59:53\r\nnewt2 = \r\n   06-May-2009 12:32:28\r\n   06-May-2009 12:59:50\r\n   06-May-2009 12:59:51\r\n   06-May-2009 12:59:52\r\n   06-May-2009 12:59:53\r\n  Name           Size            Bytes  Class         Attributes\r\n\r\n  E              1x1              3106  MException              \r\n  ans            1x3                24  double                  \r\n  d              1x1               121  datetime                \r\n  datediff       1x1               128  duration                \r\n  mydatestr      1x6                12  char                    \r\n  newdate        1x1               121  datetime                \r\n  newt           5x1               201  datetime                \r\n  newt1          1x1               147  datetime                \r\n  newt2          5x1               185  datetime                \r\n  olddate        1x1               121  datetime                \r\n  t              5x1               161  datetime                \r\n  t1             1x1               129  datetime                \r\n  tod            5x1               160  duration                \r\n\r\n<\/pre><p>So now we have decent working code, much nicer than the first successful attempt too!<\/p><h4>P.S.<a name=\"7cb6d3c7-04c2-4708-8d7b-0e043ffc278a\"><\/a><\/h4><p>This also could have been done with a for-loop instead of working on the arrays as arrays.  For a larger set of data, my colleague found that a for-loop solution (including working with the old-style <tt>datestr<\/tt> etc. tools) took 29.0 seconds, and the first vectorized version took 2.3 seconds.  I'm guessing the second attempt is faster but didn't bother timing it.<\/p><h4>And You?<a name=\"929598e9-6233-4962-8ded-889159f5b047\"><\/a><\/h4><p>Have you been using the new date and time features, starting with Release R2014b?  What have you had success or struggled with? Let us know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=1676#respond\">here<\/a>.<\/p><p>In case it helps you, we also have a <a href=\"https:\/\/www.mathworks.com\/videos\/date-and-time-arrays-95317.html\">short video<\/a> about <tt>datetime<\/tt> that you might be interested in. Happy computing!<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_3560e52c444344998cb10fbcd1e1400b() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='3560e52c444344998cb10fbcd1e1400b ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 3560e52c444344998cb10fbcd1e1400b';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2016 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_3560e52c444344998cb10fbcd1e1400b()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2016a<br><\/p><\/div><!--\r\n3560e52c444344998cb10fbcd1e1400b ##### SOURCE BEGIN #####\r\n%% Setting a New Time Basis for Date \/ Time Data\r\n% I was recently working with someone who had some temporal data in various\r\n% formats and was trying to merge them in a meaningful way.\r\n%% Problem Setup\r\n% I made a small subset of the data and loaded it in.\r\nload setTimeBasis\r\nwhos\r\n%%\r\n% First I looked at the both variables.\r\nmydatestr\r\n%%\r\nt\r\n%% \r\n% You can see that I have a string representing a date (May 6, 2009), and\r\n% then a <https:\/\/www.mathworks.com\/help\/matlab\/ref\/datetime.html\r\n% |datetime|> array in |t|.  Though I saw that |t| was a |datetime|, I kept\r\n% thinking of it as a duration since I didn't see a month, year, etc.  And\r\n% I have to confess, though I usually do check the documentation, I was\r\n% really resisting it yesterday - not sure why - perhaps I was being more\r\n% obstinate than usual :-) !\r\n%% What Didn't Work\r\n% Next I converted the base date to |datetime|, and tried adding it to |t|.\r\nd = datetime('090506','InputFormat','MMddyy')\r\ntry\r\n     d + t\r\ncatch E\r\n    disp(E.message)\r\nend\r\n%% First Working Shot\r\n% Ok, so that didn't work.  I finally realized that even though |t| was\r\n% only displaying hours, minutes, and seconds, because it was a |datetime|\r\n% array and not a <https:\/\/www.mathworks.com\/help\/matlab\/ref\/duration.html\r\n% |duration|>, I had to deal with that!  I also realized that I used the\r\n% wrong formatting for converting the original date for |mydatestr| to a\r\n% |datetime|!  What was I thinking!\r\nnewdate = datetime(mydatestr,'InputFormat','yyMMdd')\r\n%%\r\n% Let's inspect the first time point now.\r\nt1 = t(1);\r\n[year(t1) month(t1) day(t1)]\r\n%%\r\n% Aha!  Now I need to convert the date (not time) values in |t| to that in\r\n% |olddate|!  I can do this by computing the difference in dates, leading\r\n% to a |duration| array which I *can* add to the base date.\r\nolddate = datetime(year(t1),month(t1),day(t1));\r\ndatediff = newdate - olddate\r\nnewt = t + datediff\r\n%%\r\n% Let's check out what's in |newt| now (I know, |eye(newt)|!)\r\nnewt1 = newt(1);\r\nnewt1.Format = 'yy-MM-dd hh:mm:ss'\r\n\r\n%% Much Better Answer\r\n% Frankly, I got it to work but it was really ugly and somewhat contorted.\r\n% Had I responsibly read the documentation before, I just might have found\r\n% a better way.  Instead of showing you the doc right now, I will show you\r\n% another way to poke for more information.  Let's find out what I can do\r\n% with a |datetime| array.\r\nmethods(t)\r\n%%\r\n% Hmmmm!  |timeofday| sure sounds interesting, doesn't it!?!  So let's try\r\n% again.\r\nnewdate = datetime(mydatestr,'InputFormat','yyMMdd')\r\ntod = timeofday(t)\r\nnewt2 = newdate + tod\r\nwhos\r\n%%\r\n% So now we have decent working code, much nicer than the first successful\r\n% attempt too!\r\n%% P.S.\r\n% This also could have been done with a for-loop instead of working on the\r\n% arrays as arrays.  For a larger set of data, my colleague found that a\r\n% for-loop solution (including working with the old-style |datestr| etc.\r\n% tools) took 29.0 seconds, and the first vectorized version took 2.3\r\n% seconds.  I'm guessing the second attempt is faster but didn't bother\r\n% timing it.\r\n\r\n%% And You?\r\n% Have you been using the new date and time features, starting with Release\r\n% R2014b?  What have you had success or struggled with? Let us know\r\n% <https:\/\/blogs.mathworks.com\/loren\/?p=1676#respond here>.\r\n%\r\n% In case it helps you, we also have a\r\n% <https:\/\/www.mathworks.com\/videos\/date-and-time-arrays-95317.html short\r\n% video> about |datetime| that you might be interested in. Happy computing!\r\n\r\n##### SOURCE END ##### 3560e52c444344998cb10fbcd1e1400b\r\n-->","protected":false},"excerpt":{"rendered":"<!--introduction--><p>I was recently working with someone who had some temporal data in various formats and was trying to merge them in a meaningful way.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2016\/06\/13\/setting-a-new-time-basis-for-date-time-data\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[57,6],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1676"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=1676"}],"version-history":[{"count":5,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1676\/revisions"}],"predecessor-version":[{"id":1691,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1676\/revisions\/1691"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=1676"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=1676"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=1676"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}