Comments on: Using MATLAB to Grade

By: Peter Perkins

Peter Perkins — Mon, 09 Mar 2009 13:59:03 +0000

Sung, the Statistics Toolbox has functions such as NANMEAN and NANSTD with “NAN” explicitly in the name, and those all treat NaNs as “missing values” and remove them. But many other Statistics Toolbox functions also treat NaN as a missing value flag, even if “NAN” is not in the name.

By: Sung Soo

Sung Soo — Fri, 06 Mar 2009 20:51:36 +0000

Honestly, I welcome this ‘dataset’ feature. BTW when was this introduced? I haven’t been aware of it.

At first glance, it really looks like the basic data type in ‘R statistics language’. Though R language is (in my personal opinion) not modern at all and unintuitive, it has a great advantage on handling data. It is mainly because of its data structure, which is almost identical to ‘dataset’ introduced in this blog.

Another advantage of R is its huge user base (most of them are previous users of SAS or SPSS), and their contribution to R with so many tools.

I don’t want R’s not-modern programming style to creep into MATLAB’s statistical toolbox, but I really hope MATLAB can deal with essential features of R language. ‘dataset’ looks a very good thing to be added. The next thing I want from MATLAB is to provide broad range of essential functions that properly deal ‘NaN’, which is a missing data. If most functions of statistical toolbox has its ‘NaN’ version, it will be great help to most statistician.

By: Peter Perkins

Peter Perkins — Thu, 04 Sep 2008 12:45:35 +0000

Jasmine, I’m not exactly sure of the situation that you’re describing. There’s no reason to preallocate an array in MATLAB _just_ because it’s big. However, it is good practice to preallocate an array if, for example, you are going to fill it in one row at a time in a loop, especially if it will be big. So I’m guessing you mean something like, “I will be storing rows of data one a time and end up with a large dataset array, so I want to preallocate it.”

The way to do that is more or less just as with any other array: create variables using, for example, ZEROS, and create a dataset array from those. Then overwrite each row as you get the real data.

For categorical variables, it’s hard to say what your code should look like without knowing what data you are converting, but it will definitely be advantageous to “pre-define” the levels you care about, using the third input to the NOMINAL/ORDINAL constructor.

Hope this helps.

By: jasmine

jasmine — Wed, 03 Sep 2008 20:31:22 +0000

Hi Loren,

I am trying to store both numerical and categorical values to a dataset array. As the data size is big, I wan to initialize the dataset array so that I can speed up the operation. How can I do that?

Many thanks!

By: Jessee

Jessee — Tue, 05 Aug 2008 12:20:25 +0000

Peter, I suppose the data I typically use is homogeneous in the sense that the columns are all doubles. You’re right though, the dataset array is better for mixed data types.

By: Dimitri Shvorob

Dimitri Shvorob — Wed, 30 Jul 2008 16:50:32 +0000

Kudos to Loren for highlighting this neat (relatively) new feature of Statistics Toolbox. I hope dataset arrays’ functionality will be expanding in forthcoming releases.

By: Peter Perkins

Peter Perkins — Thu, 24 Jul 2008 15:29:50 +0000

Jessee, there is a property that you can use to tag variables with units. For example,

>> load hospital % sample file in Statistics Toolbox
>> hospital.Properties.Units
ans =
” ” ‘Yrs’ ‘Lbs’ ” ‘mm Hg’ ‘Counts’
>> hospital.Weight = hospital.Weight/2.2;
>> hospital.Properties.Units{4} = ‘kg’;

The units also show up in the description of each dataset variable if you use the summary method on a dataset array. Note that these are just for the purpose of labelling, there is not any provision for conversions or units checking in math or anything like that.

I’m curious about your comment about large datasets. Certainly if you have data that are homogeneous, you are better off using a matrix. But if your columns have different types, the dataset array is every bit as efficient as a scalar structure, and a good deal more efficient (memory-wise) than a structure array that has one element for every row of your data. It has the benefit over a scalar structure that you can easily subscript across dataset variables, which correspond to fields for the scalar atructure solution. And it allows you to use names for both dataset variables and observations.

By: Jessee

Jessee — Thu, 24 Jul 2008 12:55:40 +0000

I could potentially see myself using dataset for casually looking at data, but from an application standpoint where you might be processing large data sets I think I’d stick with structures and matrices.

Is there any way to associate units with a column in the data set?