Working with Text in MATLAB

I'd like to introduce today's guest blogger, Dave Bergstein, a MATLAB Product Manager at MathWorks. In today's post, Dave discusses recent updates to text processing with MATLAB.

Contents

In today's post I share a text processing example using the new string array and a collection of new text manipulation functions, both introduced in R2016b. I also give recommendations on when best to use string, char, or cell for text and share some of our thinking on the future.

Also be sure to check out Toshi's post Introducing String Arrays and Loren's post Singing the Praises of Strings.

Example: How Late Is My Bus?

My friend in New York City talks about the delays on her bus route. Let's look at some data to see what typical delays are for trains and buses. The Open NY Initiative shares data which includes over 150,000 public transit events spanning 6 years. I downloaded this data as a CSV file from: https://data.ny.gov/Transportation/511-NY-MTA-Events-Beginning-2010/i8wu-pqzv

Import the Data

I read the data into a table using readtable and specify the TextType name-value pair as string to read the text as string arrays.

data = readtable('511_NY_MTA_Events__Beginning_2010.csv','TextType','string');

Warning: Variable names were modified to make them valid MATLAB
identifiers. The original names are saved in the VariableDescriptions
property. 

Here is a list of variables in the table:

data.Properties.VariableNames

ans =
  1×13 cell array
  Columns 1 through 4
    'EventType'    'OrganizationName'    'FacilityName'    'Direction'
  Columns 5 through 9
    'City'    'County'    'State'    'CreateTime'    'CloseTime'
  Columns 10 through 13
    'EventDescription'    'RespondingOrganiz…'    'Latitude'    'Longitude'

data.EventDescription is a string array which contains the event descriptions. Let's take a closer look at the events.

eventsStr = data.EventDescription;

Unlike character vectors or cell array of character vectors, each element of the string array is a string itself. See how I can index the string array just as I would a numeric array and get strings arrays back.

eventsStr(1:3)

ans = 
  3×1 string array
    "MTA NYC Transit Bus: due to Earlier flooding Q11 into Hamilton Beach normal service resumed"
    "MTA NYC Transit Bus: due to Construction, northbound M1 Bus area of 147th Street:Adam Clayton Powell Junior"
    "MTA NYC Transit Subway: due to Delays, Bronx Bound # 2 & 3 Lines at Nevins Street Station (Brooklyn)"

Many of the event descriptions report delays like 'operating 10 minutes late'. See for example how the 26-minute delay is reported in event 5180.

eventsStr(5180)

ans = 
    "MTA Long Island Rail Road: due to Debris on tracks, westbound Montauk Branch between Montauk Station (Suffolk County)  and Jamaica Station (Queens)  The 6:44 AM from Montauk due Jamaica at 9:32 AM, is operating 26 minutes late due to an unauthorized vehicle on the tracks near Hampton Bays."

Identify Delays

I want to find all the events which contain ' late '. MATLAB R2016b also introduced more than a dozen new functions for working with text. These functions work with character vectors, cell arrays of character vectors, and string arrays. You can learn about these functions from the characters and strings page in our documentation.

I convert the text to all lowercase and determine which events contain ' late ' using the contains function.

eventsStr = lower(eventsStr);
idx = contains(eventsStr,' late ');
lateEvents = eventsStr(idx);

Extract the Delay Times

I extract the minutes late from phrases like 'operating 10 minutes late' using the functions extractAfter and extractBefore.

Let's look at the first late event. The exact phrase we are seeking doesn't appear in this event. When we look for the text following 'operating' we get back a missing string.

lateEvents(1)
extractAfter(lateEvents(1),'operating')

ans = 
    "mta long island rail road: due to delays, westbound babylon branch between speonk station (speonk)  and new york penn station (manhattan)  the 5:08 a.m. departure due ny @ 7:02 a.m. is 15 minutes late @ babylon."
ans = 
    <missing>

Let's look at the second late event. This string contains the phrase 'operating 14 minutes late'. Extracting the text after 'operating' we get '14 minutes late due to signal problems'. Extracting the text before 'minutes late' we get back ' 14 ' which we can convert to a numeric value using double.

lateEvents(2)
s = extractAfter(lateEvents(2),'operating')
s = extractBefore(s,'minutes late')
minLate = double(s)

ans = 
    "mta long island rail road: due to delays westbound ronkonkoma branch out of bethpage station (suffolk county) the 8:01 am train due into penn station at 8:47 am is operating 14 minutes late due to signal problems"
s = 
    " 14 minutes late due to signal problems"
s = 
    " 14 "
minLate =
    14

Success! We extracted the train delay from the event description. Now let's put this all together. I extract the minutes late from all the events and drop the missing values using the rmmissing function. I then convert the remaining values to numbers using double and plot a histogram of the results.

s = extractAfter(lateEvents,'operating');
s = extractBefore(s,'minutes late');
s = rmmissing(s);
minLate = double(s);

histogram(minLate,0:5:40)
ylabel('Number of Events')
xlabel('Minutes Late')
title({'Transit Delays','NY Metropolitan Transit Authority'})

It looks like reported delays are often 10-15 minutes. This simple routine captures many of the transit delays, but not all. The pattern doesn't always fit (consider again lateEvents(1)). I also left out any delays that may be reported in hours. Can you improve it?

Text as Data

String arrays are a great choice for text data like the example above because they are memory efficient and perform better than cell arrays of character vectors (previously known as cellstr).

Let's compare the memory usage. I convert the string array to a cell array of character vectors with the cellstr command and check the memory with whos. See the Bytes column - it shows the string array is about 12% more efficient.

eventsCell = cellstr(eventsStr);
whos events*

  Name                 Size               Bytes  Class     Attributes

  eventsCell      151225x1             73208886  cell                
  eventsStr       151225x1             64662486  string              

The memory savings can be much greater for many smaller pieces of text. for example, suppose I want to store each word as a separate array element. First I join all 150,000 reports into a single long string using the join function. I then split this long string on spaces using the split function. The result is a string array storing over 4 million words in separate elements. Here the memory savings is nearly 2X.

wordsStr = split(join(eventsStr));
wordsCell = split(join(eventsCell));
whos words*

  Name                 Size                Bytes  Class     Attributes

  wordsCell      4356256x1             535429652  cell                
  wordsStr       4356256x1             284537656  string              

String arrays also perform better. You can achieve the best performance using string arrays in combination with the text manipulation functions introduced in R2016b. Here I compare the performance of replace on a string array with that of strrep on a cell array of character vectors. See how replace with a string array is about 4X faster than strrep with a cell array.

f1 = @() replace(eventsStr,'delay','late');
f2 = @() strrep(eventsCell,'delay','late');
timeit(f1)
timeit(f2)

ans =
     0.062507
ans =
      0.23239

Recommendations on Text Type

So, should you use string arrays for all your text? Maybe not yet. MATLAB has three different ways to store text:

For now (as of R2017a), we encourage you to use string arrays to store text data such as the transit events. We don’t recommend using string arrays elsewhere yet since string arrays aren’t yet accepted everywhere in MATLAB. Notice how I used a character vector for specifying the filename in readtable and a cell array of character vectors for the figure title.

Looking to the Future

What about in the future? We feel string arrays provide a better experience than character vectors and cell arrays of character vectors. Our plan is to roll out broader use of string arrays over time.

In the next few releases we will update more MATLAB functions and properties to accept string arrays in addition to character vectors and cell arrays of character vectors. As we do so, it will become easier for you to use string arrays in more places.

Next we will replace cell arrays of character vectors in MATLAB with string arrays. Note that cell arrays themselves aren't going anywhere. They are an important MATLAB container type and good for storing mixed data types or arrays of jagged size among other uses. But we expect their use for text data will diminish and become largely replaced by string arrays which are more memory efficient and perform better for pure text data.

Beyond that, over time, we will use string arrays in new functions and new properties in place of character vectors (but will continue returning character vectors in many places for compatibility). We expect character vectors will continue to live on for version-to-version code compatibility and special use cases.

Speaking of compatibility: we care deeply about version-to-version compatibility of MATLAB code, today more than ever. So, we are taking the following steps in our roll out of string arrays:

d = datetime('now');
f = {'h','m','s'};   % use a cell array
for n = 1:3,
    d.Format = f{n};
    disp(d)
end

   9
   43
   10

f = ["h","m","s"];   % use a string array
for n = 1:3,
    d.Format = f{n};
    disp(d)
end

   9
   43
   10

Expect to hear more from me on this topic. And please share your input with us by leaving a comment below. We're interested to hear from you.

We hope string arrays will help you accomplish your goals and that the steps we're taking provide a smooth adoption. If you haven't tried string arrays yet, learn more from our documentation on characters and strings.

Get the MATLAB code

Published with MATLAB® R2017a