Reading Structured Data Files

Posted by Jiro Doke, January 21, 2011

23 views (last 30 days) | 0 Likes | 0 comment

Jiro's pick this week is NRGSymphonieRead by a fellow application engineer, Todd Schultz, who sits down the hall from me.

Have you ever needed to bring data into MATLAB from a structured data file, but found that the structure is a bit too complex for the basic file I/O functions to work out of the box? In those cases, you typically have to write a reader function for your particular type of file. Todd wrote this function to read text files produced from an NRG Symphonie data logger. But he used techniques that can be applied to work with other types of files. As long as there is some structure to the file, it is possible to write a reader for it, and Todd's file demonstrates how to do so.

In his entry, he included a sample file to test his code with. The file looks like this:

   SDR	6.0.14

   -----Logger Information-----
   Model #	3090
   Serial #	09351
   Hardware Rev.	011-010-000

   -----Site Information-----
   Site #	1029
   Site Desc	New Site
   Project Code	New
   Project Desc	New Project
   Site Location
   Site Elevation	41
   Latitude	N 042° 14.348'
   Longitude	W 070° 51.088'
   Time offset (hrs)	-5

   -----Sensor Information-----

   Channel #	1
   Type	1
   Description	NRG #40 Anem. m/s
   Details
   Serial Number	SN:
   Height	49   m
   Scale Factor	0.765
   Offset	0.35
   Units	m/s

   Channel #	2
   Type	1
   Description	NRG #40 Anem. m/s
   Details
   Serial Number	SN:
   Height	49   m
   Scale Factor	0.765
   Offset	0.35
   Units	m/s

      .
      .
      .

   Date & Time Stamp	CH1Avg	CH1SD	CH1Max	CH1Min	...
   05/25/2007 11:00:00	0.3500	0.0000	0.3500	0.3500	...
   05/25/2007 11:10:00	0.3500	0.0000	0.3500	0.3500	...
   05/25/2007 11:20:00	1.7473	1.1298	4.1707	0.3500	...

      .
      .
      .

As you can see, it contains some header information, grouped by descriptive titles ("Logger Information", "Site Information", "Sensor Information"), and then a block of data in a tabular form. Todd's function scans the file for the header information, puts it into a structure variable, and extracts the tabular data into a dataset array. His code is very well commented and broken up into sections that handle different parts of the file. Anyone wanting to learn how to write a similar reader for their particular data files should take a look at Todd's file. I found it helpful when I ran this in debug mode. I could see what each step was doing and how the various data was being extracted.

Comments

Tell us here about any file reader functions you've written. You can also leave a comment for Todd.

Published with MATLAB® 7.11