Comments on: A Way to Account for Missing Data

By: Parul

Parul — Fri, 02 Sep 2011 17:10:12 +0000

hi,

thnx a lot….. its amazing code.
it has saved mine hell lot of time… :)

rly thnx…..

By: Loren

Loren — Fri, 05 Feb 2010 18:40:26 +0000

Eleanor-

You either need an algorithm to determine which 0s should be replaced by NaN values or you will need to pre-edit the data yourself.

–Loren

By: Eleanor

Eleanor — Fri, 05 Feb 2010 18:34:38 +0000

Loren-
I have a different sort of problem with NaNs and 0s. My data has zeros where data is missing, but also has zeros that are real values. Having the missing values as zeros skews my correlations. I want to replace the zeros with NaNs, but only in columns where zeros can’t be real values.

By: neuro11

neuro11 — Tue, 21 Jul 2009 12:18:30 +0000

hi…Loren thanx for the article,it was quite helpful for me.I was handling some data where i needed a NAN replacement..so nansum type function was not helpful.

By: Loren

Loren — Tue, 10 Mar 2009 12:14:20 +0000

Chris-

MATLAB is smart enough to not convert the logicals to doubles in their entirety before doing the summation. You can convince yourself on windows by watching the task manager as you perform the operation on a large enough array.

–Loren

By: Chris Rodgers

Chris Rodgers — Tue, 10 Mar 2009 03:59:47 +0000

Is a sum over logicals, as in howMany=sum(isnan(DataMatrix)), optimized for speed? It seems inefficient to convert logicals to doubles and then add, especially since it already had to iterate through the entire matrix to do the isnan check.

By: Tim Davis

Tim Davis — Fri, 26 Oct 2007 02:10:32 +0000

Replacing NaN’s with zeros prior to heavy-duty computation is a good thing (assuming that it still gives you the right answer of course). NaN’s, Inf’s and the like cause the floating-point hardware to slow *way* down. Try this:

A=rand(2000);
B=rand(2000);
C=nan(2000);
tic; E = A+B ; toc
tic; D = A+C ; toc

The first computation takes 1.3 seconds on my Pentium 4 desktop in MATLAB 7.5. The 2nd takes 0.1 seconds. So NaN’s are great when used carefully, but keep an eye on performance if MATLAB seems sluggish when you abuse them.

By: Matt G

Matt G — Sat, 20 Oct 2007 03:09:14 +0000

I run into this exact problem all the time. A few months ago I posted a function in the File Exchange named “ignorenan”. It also uses the accumarray function but handles n-d data and has an input to specify which dimension to operate on. Hopefully someone else may find it useful…

By: Doug Hull

Doug Hull — Fri, 12 Oct 2007 17:53:51 +0000

I covered the uses of NaN in graphics in a movie on my blog earlier this week:

http://blogs.mathworks.com/pick/2007/10/08/matlab-basics-video-using-nan-as-placeholder-data-in-graphics/

Doug

By: Duane Hanselman

Duane Hanselman — Fri, 12 Oct 2007 16:20:29 +0000

For those looking for a partial solution without the stats toolbox, there is #10235 on the file exchange. It demonstrates what Loren illustrates here for all common stat measures.