{"id":5095,"date":"2014-01-17T09:00:55","date_gmt":"2014-01-17T14:00:55","guid":{"rendered":"https:\/\/blogs.mathworks.com\/pick\/?p=5095"},"modified":"2017-03-27T20:32:20","modified_gmt":"2017-03-28T00:32:20","slug":"scraping-data-from-the-web","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/pick\/2014\/01\/17\/scraping-data-from-the-web\/","title":{"rendered":"Scraping data from the web"},"content":{"rendered":"

Jiro<\/a>'s pick this week is urlfilter<\/tt><\/a> by Ned Gulley<\/a>.<\/p>

Many of you may know Ned from various parts of MATLAB Central<\/a>, such as the community blog \"MATLAB Spoken Here\"<\/a>. If you're a frequent visitor of MATLAB Central, you may have also visited Trendy, which allows you to quickly query and plot trends from the web. One of the utility functions provided within Trendy has been urlfilter<\/tt>, and it's a convenient function that allows you to easily scrape data from a web page. Now, you can use urlfilter<\/tt> outside of Trendy!<\/p>

To see how it works, take a look at the Trendy tutorial or the published example script included with Ned's entry. But here's a quick example of how it could be used.<\/p>

Let's say that I want to grab and plot the high and low temperatures in Natick, MA for the next 10 days. I will grab data from this URL<\/a> at http:\/\/www.wunderground.com<\/a>. As you can see from the web page, the 10-day forecast is displayed about halfway down the page in a table. Each day has a header in the format of \"day of week, day\", e.g. \"Friday, 17\".<\/p>

First, I calculate the days I'm interested in, which is today to 10 days from today. I also determine the day of the week using the weekday<\/tt><\/a> function. I need this information, because urlfilter<\/tt> will use this to scrape the necessary data.<\/p>

days = floor(now):floor(now)+9;\r\n[~, ~, dayval] = datevec(days);\r\n[~, weekdaystr] = weekday(days, 'long'<\/span>);\r\n<\/pre>
Now, I simply use urlfilter<\/tt> to iterate through each day as the search term.<\/p>
% Pre-allocate variables<\/span>\r\nlow = nan(1,length(days));\r\nhigh = nan(1,length(days));\r\n\r\nurl = 'http:\/\/www.wunderground.com\/q\/zmw:01760.1.99999'<\/span>;\r\nfor<\/span> iD = 1:length(days)\r\n    % Search term<\/span>\r\n    str = [strtrim(weekdaystr(iD,:)), ', '<\/span>, num2str(dayval(iD))];\r\n    disp(['Scraping temperatures for \"'<\/span>, str, '\"...'<\/span>])\r\n\r\n    % Fetch 2 values (high and low)<\/span>\r\n    vals = urlfilter(url,str,2);\r\n\r\n    high(iD) = vals(1);\r\n    low(iD) = vals(2);\r\nend<\/span>\r\n<\/pre>
Scraping temperatures for \"Friday, 17\"...\r\nScraping temperatures for \"Saturday, 18\"...\r\nScraping temperatures for \"Sunday, 19\"...\r\nScraping temperatures for \"Monday, 20\"...\r\nScraping temperatures for \"Tuesday, 21\"...\r\nScraping temperatures for \"Wednesday, 22\"...\r\nScraping temperatures for \"Thursday, 23\"...\r\nScraping temperatures for \"Friday, 24\"...\r\nScraping temperatures for \"Saturday, 25\"...\r\nScraping temperatures for \"Sunday, 26\"...\r\n<\/pre>
Let's plot the results. To show the temperature in two different units, I'm using my plot2axes<\/tt><\/a> (shameless plug).<\/p>
ax = plot2axes(days,high,'r.-'<\/span>,days,low,'b.-'<\/span>, ...<\/span>\r\n    'YScale'<\/span>,@(x)5\/9*(x-32));\r\nylabel(ax(1),'Temperature (\\circF)'<\/span>)\r\nylabel(ax(2),'Temperature (\\circC)'<\/span>)\r\ndatetick('x'<\/span>,'mmm dd'<\/span>,'keepticks'<\/span>)\r\nlegend('High'<\/span>,'Low'<\/span>)\r\ntitle('10-day Forecast for Natick, MA, U.S.A'<\/span>)\r\n<\/pre>\"\" 
Note that I could have done this more efficiently with a single call to urlfilter<\/tt>, extracting about 40 numbers at once, and then parsing the numbers to get the necessary high and low temperatures. I used the above approach to make it easier to understand.<\/p>
Comments<\/b><\/p>
Wasn't that easy? Give this a try, and let us know what you think here<\/a> or leave a comment<\/a> for Ned. If you find interesting data, consider tracking the trend using Trendy<\/a>!<\/p>