Comments on: Read data from the web with URLREAD http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/ Doug Hull is a proud MathWorker who is on a mission to help you with MATLAB. Fri, 10 Feb 2012 20:31:50 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Ritesh http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2941 Ritesh Sun, 11 Dec 2011 15:35:06 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2941 Hi, I want to read data(download an excel file) from a website. The data is not open source. I have to log in first with password and then only I can access data through a web browser. How can I download the same file through urlread command? Hi,

I want to read data(download an excel file) from a website. The data is not open source. I have to log in first with password and then only I can access data through a web browser.

How can I download the same file through urlread command?

]]> By: dhull http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2438 dhull Tue, 09 Nov 2010 19:13:03 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2438 Thisara, Can you read the files if they were local? Can you use the FTP command to bring them local? Doug Thisara,

Can you read the files if they were local? Can you use the FTP command to bring them local?

Doug

]]> By: Thisara http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2435 Thisara Tue, 09 Nov 2010 00:01:09 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2435 Hi Doug, Thanks for the nice piece of work. I am trying to download some netcdf data from web. I tried to use urlread, but couldn't success.File can not be scanned as you did. But if I can download the netcdf file, I can read it through Matlab in my computer. Is there any other way to download the netcdf file through Matlab ? Cheers Thisara this is the location of data : http://opendap-tpac.arcs.org.au/thredds/catalog/library/argo_australia/aoml/5900607/profiles/catalog.html?dataset=library/argo_australia/aoml/5900607/profiles/D5900607_001.nc Hi Doug,
Thanks for the nice piece of work. I am trying to download some netcdf data from web. I tried to use urlread, but couldn’t success.File can not be scanned as you did.
But if I can download the netcdf file, I can read it through Matlab in my computer.

Is there any other way to download the netcdf file through Matlab ?

Cheers
Thisara

this is the location of data :
http://opendap-tpac.arcs.org.au/thredds/catalog/library/argo_australia/aoml/5900607/profiles/catalog.html?dataset=library/argo_australia/aoml/5900607/profiles/D5900607_001.nc

]]> By: dhull http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2038 dhull Tue, 27 Apr 2010 18:20:17 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2038 @Paul, I like the following: <pre class="code"> urlwrite('http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1','data.csv'); [num, txt] = xlsread('data.csv'); </pre> You now have num and txt. These can be parsed pretty easily, especially if the format of all of these are always the same. I actually like pulling the data locally into a file and then reading it in. Something about having the file close by, even if in a temporary file that you toss later feels right. -Doug @Paul,

I like the following:


urlwrite('http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1','data.csv');
[num, txt] = xlsread('data.csv');

You now have num and txt. These can be parsed pretty easily, especially if the format of all of these are always the same.

I actually like pulling the data locally into a file and then reading it in. Something about having the file close by, even if in a temporary file that you toss later feels right.

-Doug

]]> By: paul http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2037 paul Tue, 27 Apr 2010 15:28:16 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2037 To those asking about converting the string data to matrix data, str2double worked for me. Somewhere somebody recommends to save the string as a csv and then read the csv back into Matlab, but this seems very roundabout to me. To those asking about converting the string data to matrix data, str2double worked for me. Somewhere somebody recommends to save the string as a csv and then read the csv back into Matlab, but this seems very roundabout to me.

]]> By: paul http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2036 paul Mon, 26 Apr 2010 21:58:16 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-2036 Thanks Doug, great tutorial. I'm trying to do something similar, which I know is possible, but just a bit beyond my matlab ability. I am trying to extract historical wind data from a website, and while it is comma separated, it is a mixture of words, numbers, and annotations. I can get the data into matlab, but I can't figure out how to (1) save the data as a csv file for archiving, (2) extract specific parts of the block (like hour), and (3) turn the data into variables (double?) on which I can do math. I have found several possible solutions but keep getting stuck on one part or another. I think I could do it in a way that would require a ton of loops and if/then statements, but since I want to mine data from many websites, this would likely get slow. Any help would be much appreciated. THANK YOU!! Here's my start: block=urlread('http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1'); data=textscan(block,'%s %s %s %s %s %s %s %s %s %s %s %s','delimiter',','); urwrite('http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1',data.csv'); data = csvread('data.csv'); data2=textscan(block,'%s %s','delimiter',':'); clear block points=length(data{1}); dataTime=data{1}(2:points-1); dataTemperature=data{2}(2:points-1); dataHumidity=data{3}(2:points-1); dataWindDirection=data{7}(2:points-1); dataWindSpeedAvg=data{8}(2:points-1); dataWindSpeedGust=data{9}(2:points-1); dataHour=data2{1}(2:points-1); Thanks Doug, great tutorial. I’m trying to do something similar, which I know is possible, but just a bit beyond my matlab ability. I am trying to extract historical wind data from a website, and while it is comma separated, it is a mixture of words, numbers, and annotations. I can get the data into matlab, but I can’t figure out how to (1) save the data as a csv file for archiving, (2) extract specific parts of the block (like hour), and (3) turn the data into variables (double?) on which I can do math. I have found several possible solutions but keep getting stuck on one part or another. I think I could do it in a way that would require a ton of loops and if/then statements, but since I want to mine data from many websites, this would likely get slow. Any help would be much appreciated. THANK YOU!! Here’s my start:

block=urlread(‘http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1‘);
data=textscan(block,’%s %s %s %s %s %s %s %s %s %s %s %s’,'delimiter’,',’);

urwrite(‘http://www.wunderground.com/history/airport/KSVC/2010/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA&format=1',data.csv‘);
data = csvread(‘data.csv’);

data2=textscan(block,’%s %s’,'delimiter’,':’);

clear block
points=length(data{1});
dataTime=data{1}(2:points-1);
dataTemperature=data{2}(2:points-1);
dataHumidity=data{3}(2:points-1);
dataWindDirection=data{7}(2:points-1);
dataWindSpeedAvg=data{8}(2:points-1);
dataWindSpeedGust=data{9}(2:points-1);

dataHour=data2{1}(2:points-1);

]]> By: dhull http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1877 dhull Mon, 22 Feb 2010 21:58:34 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1877 @Need Help The command does return the data as one big string. When you say: "I was wondering if there is a function which converts it into a string" What do you mean? What would a typical return from URLREAD() look like so we know what you are getting and what would the desired output look like? Doug @Need Help

The command does return the data as one big string. When you say:

“I was wondering if there is a function which converts it into a string”

What do you mean?

What would a typical return from URLREAD() look like so we know what you are getting and what would the desired output look like?

Doug

]]> By: Need Help http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1876 Need Help Mon, 22 Feb 2010 21:54:05 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1876 Hi Doug, I have a question. I want to know a way to convert the string you obtain from using urlread() into a matrix. In the case scenario described above, the data is in columns already but I was wondering if there is a function which converts it into a string without the requirement of the data being in columns already. Please let me know. Thanks Hi Doug, I have a question. I want to know a way to convert the string you obtain from using urlread() into a matrix. In the case scenario described above, the data is in columns already but I was wondering if there is a function which converts it into a string without the requirement of the data being in columns already. Please let me know. Thanks

]]> By: dhull http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1668 dhull Wed, 11 Nov 2009 19:49:48 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1668 Looks like this was answered in the newsgroup. Looks like this was answered in the newsgroup.

]]> By: Magnus http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1667 Magnus Wed, 11 Nov 2009 15:57:15 +0000 http://blogs.mathworks.com/videos/2009/03/03/read-data-from-the-web-with-urlread/#comment-1667 Hi Doug, I tried to use you method to import data from the web, but it did not work. I did post a link on matlabcentral, http://www.mathworks.in/matlabcentral/newsreader/view_thread/265578, do you have any suggestion how I should read the data from the web and put it into a matrix? -Magnus Hi Doug,

I tried to use you method to import data from the web, but it did not work. I did post a link on matlabcentral, http://www.mathworks.in/matlabcentral/newsreader/view_thread/265578, do you have any suggestion how I should read the data from the web and put it into a matrix?

-Magnus

]]>