{"id":818,"date":"2013-12-09T10:44:00","date_gmt":"2013-12-09T15:44:00","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=818"},"modified":"2013-12-02T10:45:02","modified_gmt":"2013-12-02T15:45:02","slug":"getting-data-from-a-web-api-in-parallel","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2013\/12\/09\/getting-data-from-a-web-api-in-parallel\/","title":{"rendered":"Getting Data from a Web API in Parallel"},"content":{"rendered":"\r\n<div class=\"content\"><!--introduction--><p>I'd like to introduce this week's guest blogger <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/answers\/contributors\/1287414-edric-ellis\">Edric Ellis<\/a>. Edric works for the Parallel Computing development team here at The MathWorks. In this post he will talk about using the <tt>parfeval<\/tt> command in Parallel Computing Toolbox.<\/p><p>In release R2013b, we introduced a new function <a href=\"https:\/\/www.mathworks.com\/help\/distcomp\/parfeval.html\"><tt>parfeval<\/tt><\/a> to Parallel Computing Toolbox which allows you to run MATLAB&reg; functions on workers in a parallel pool without blocking your desktop MATLAB. This allows you to create parallel programs that are more loosely structured than those using either <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/parfor.html\"><tt>parfor<\/tt><\/a> or <a href=\"https:\/\/www.mathworks.com\/help\/distcomp\/spmd.html\"><tt>spmd<\/tt><\/a>. You can also have your desktop MATLAB process the results as they become available, allowing you to create more interactive parallel programs.<\/p><p>We're going to take a look at using <tt>parfeval<\/tt> to speed up accessing a web service by making multiple requests in parallel, and then displaying each result as soon as it is available. We're going to query the <a href=\"http:\/\/www.flickr.com\">Flickr<\/a> image sharing site for astronomical images that have structured data associated with them. We'll approach the problem in three phases:<\/p><div><ol><li>Query the web service for relevant images<\/li><li>Use <tt>parfeval<\/tt> to submit work to the parallel workers to read the images<\/li><li>Use <tt>fetchNext<\/tt> to retrieve results and then display them.<\/li><\/ol><\/div><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#0529645e-7f17-47d9-b40d-1f1cbbbe9321\">Query the web service for relevant images<\/a><\/li><li><a href=\"#54b65ad5-3a83-4cfa-b4ac-4fac094c3a46\">Build the search URL and interpret the XML response<\/a><\/li><li><a href=\"#04c9408d-8971-4439-9b0b-917fe374c104\">Define a function to retrieve and process the photos<\/a><\/li><li><a href=\"#a7a8f5d5-fe62-46dc-9bae-ceb9edf41d77\">Use <tt>parfeval<\/tt> to submit work to the parallel workers to read the images<\/a><\/li><li><a href=\"#aeb6e231-868b-4cc7-a046-d172e0826873\">Prepare a figure for displaying the results<\/a><\/li><li><a href=\"#fe8c2edf-9e8d-45a1-8424-abfc8a1fd429\">Use <tt>fetchNext<\/tt> to retrieve results and then display them<\/a><\/li><li><a href=\"#410d2da5-858f-4490-9051-6493a85bad97\">Conclusions<\/a><\/li><\/ul><\/div><h4>Query the web service for relevant images<a name=\"0529645e-7f17-47d9-b40d-1f1cbbbe9321\"><\/a><\/h4><p>Flickr provides a web-based API that allows you to query its large database of images. To use the API, you submit a structured query to their servers, and an XML document is returned to you. Retrieving the resulting image data from Flickr takes some time, and this can run more quickly by performing the retrievals in parallel. This is an example of an <a href=\"http:\/\/en.wikipedia.org\/wiki\/I\/O_bound\">\"I\/O-bound\"<\/a> operation.<\/p><p>The Flickr web API is fully documented <a href=\"http:\/\/www.flickr.com\/services\/api\/\">here<\/a>. To use the API, you need a Flickr account, and you need to <a href=\"http:\/\/www.flickr.com\/services\/apps\/create\/\">request an 'API Key'<\/a> which is included in each request to the API. Once you've got your own key, you can place it in a function called <tt>flickrApiKey<\/tt> which simply returns the key as a string.<\/p><pre class=\"codeinput\">appKey = flickrApiKey;\r\n<\/pre><h4>Build the search URL and interpret the XML response<a name=\"54b65ad5-3a83-4cfa-b4ac-4fac094c3a46\"><\/a><\/h4><p>We're going to search for images which have a specific tag: <a href=\"http:\/\/www.flickr.com\/photos\/tags\/astrometrydotnet%3Astatus%3Dsolved\/\"><tt>astrometrydotnet:status=solved<\/tt><\/a>. These images are submitted to a Flickr group, and then <a href=\"http:\/\/astrometry.net\/\">astrometry.net<\/a> analyses and annotates the images. We build a search url, and then let <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/xmlread.html\"><tt>xmlread<\/tt><\/a> read the results from that URL.<\/p><pre class=\"codeinput\">url = [<span class=\"string\">'http:\/\/api.flickr.com\/services\/rest\/'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">       % API Base URL<\/span>\r\n       <span class=\"string\">'?method=flickr.photos.search'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">               % API Method<\/span>\r\n       <span class=\"string\">'&amp;api_key='<\/span>, appKey, <span class=\"keyword\">...<\/span><span class=\"comment\">                          % Our API key<\/span>\r\n       <span class=\"string\">'&amp;tags=astrometrydotnet%3Astatus%3Dsolved'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">   % Tag to search<\/span>\r\n       <span class=\"string\">'&amp;license=4'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">                                 % Creative Commons<\/span>\r\n       <span class=\"string\">'&amp;sort=interestingness-desc'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">                 % Sort order<\/span>\r\n       <span class=\"string\">'&amp;per_page=12'<\/span>, <span class=\"keyword\">...<\/span><span class=\"comment\">                               % Limit results to 12<\/span>\r\n       <span class=\"string\">'&amp;media=photos'<\/span>];                                 <span class=\"comment\">% Only return photos<\/span>\r\nresponse = xmlread(url);\r\n\r\n<span class=\"comment\">% Get all the 'photo' elements from the document.<\/span>\r\nphotos    = response.getDocumentElement.getElementsByTagName(<span class=\"string\">'photo'<\/span>);\r\nnumPhotos = getLength(photos);\r\n\r\n<span class=\"comment\">% Ensure our search query return some results.<\/span>\r\n<span class=\"keyword\">if<\/span> numPhotos == 0\r\n    error(<span class=\"string\">'Failed to retrieve photos from Flickr. XML response was: %s'<\/span>, <span class=\"keyword\">...<\/span>\r\n          xmlwrite(response));\r\n<span class=\"keyword\">end<\/span>\r\n\r\n<span class=\"comment\">% Otherwise, convert the search results into a cell array of structures<\/span>\r\n<span class=\"comment\">% containing the search result information.<\/span>\r\nphotoInfo = cell(1, numPhotos);\r\n<span class=\"keyword\">for<\/span> idx = 1:numPhotos\r\n    node = photos.item(idx-1);\r\n    photoInfo{idx}  = struct(<span class=\"string\">'farm'<\/span>,   char(node.getAttribute(<span class=\"string\">'farm'<\/span>)), <span class=\"keyword\">...<\/span>\r\n                             <span class=\"string\">'server'<\/span>, char(node.getAttribute(<span class=\"string\">'server'<\/span>)), <span class=\"keyword\">...<\/span>\r\n                             <span class=\"string\">'id'<\/span>,     char(node.getAttribute(<span class=\"string\">'id'<\/span>)), <span class=\"keyword\">...<\/span>\r\n                             <span class=\"string\">'secret'<\/span>, char(node.getAttribute(<span class=\"string\">'secret'<\/span>)), <span class=\"keyword\">...<\/span>\r\n                             <span class=\"string\">'owner'<\/span>,  char(node.getAttribute(<span class=\"string\">'owner'<\/span>)));\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><h4>Define a function to retrieve and process the photos<a name=\"04c9408d-8971-4439-9b0b-917fe374c104\"><\/a><\/h4><p>For each search result, we need to run a function to retrieve the image data. The function also searches through the Flickr comments for the structured information from astrometry.net, and returns the direction the telescope was pointing. In the next stage, we'll submit multiple invocations of this function using <tt>parfeval<\/tt>.<\/p><pre class=\"language-matlab\"><span class=\"keyword\">function<\/span> [photo, location, username] = getAstrometryPhotoFromFlickr(appKey, info)\r\n<span class=\"comment\">% First, build up the photo URL as per the Flickr documentation<\/span>\r\n<span class=\"comment\">% here: &lt;http:\/\/www.flickr.com\/services\/api\/misc.urls.html&gt;<\/span>\r\nurl = sprintf(<span class=\"string\">'http:\/\/farm%s.staticflickr.com\/%s\/%s_%s.jpg'<\/span>, <span class=\"keyword\">...<\/span>\r\n              info.farm, info.server, info.id, info.secret);\r\n<span class=\"keyword\">try<\/span>\r\n    photo = imread(url);\r\n<span class=\"keyword\">catch<\/span> E\r\n    <span class=\"comment\">% Sometimes the read fails. Try one more time.<\/span>\r\n    photo = imread(url);\r\n<span class=\"keyword\">end<\/span>\r\n<span class=\"comment\">%<\/span>\r\n<span class=\"comment\">% Get the photo info to extract the username<\/span>\r\nurl = [<span class=\"string\">'http:\/\/api.flickr.com\/services\/rest\/'<\/span>, <span class=\"keyword\">...<\/span>\r\n       <span class=\"string\">'?method=flickr.photos.getInfo'<\/span>, <span class=\"keyword\">...<\/span>\r\n       <span class=\"string\">'&amp;api_key='<\/span>, appKey, <span class=\"string\">'&amp;photo_id='<\/span>, info.id];\r\nresponse = xmlread(url);\r\nowner = response.getDocumentElement.getElementsByTagName(<span class=\"string\">'owner'<\/span>);\r\n<span class=\"keyword\">if<\/span> getLength(owner) == 1\r\n    username = char(owner.item(0).getAttribute(<span class=\"string\">'username'<\/span>));\r\n<span class=\"keyword\">else<\/span>\r\n    username = <span class=\"string\">'Unknown'<\/span>;\r\n<span class=\"keyword\">end<\/span>\r\n<span class=\"comment\">%<\/span>\r\n<span class=\"comment\">% Next, look through the comments for the photo to extract<\/span>\r\n<span class=\"comment\">% the annotation from astrometry.net.<\/span>\r\nurl = [<span class=\"string\">'http:\/\/api.flickr.com\/services\/rest\/'<\/span>, <span class=\"keyword\">...<\/span>\r\n       <span class=\"string\">'?method=flickr.photos.comments.getList'<\/span>, <span class=\"keyword\">...<\/span>\r\n       <span class=\"string\">'&amp;api_key='<\/span>, appKey, <span class=\"string\">'&amp;photo_id='<\/span>, info.id];\r\nresponse = xmlread(url);\r\n<span class=\"comment\">%<\/span>\r\n<span class=\"comment\">% Get all the actual comment elements from the XML response.<\/span>\r\ncomments = response.getDocumentElement.getElementsByTagName(<span class=\"string\">'comment'<\/span>);\r\n<span class=\"comment\">%<\/span>\r\n<span class=\"comment\">% Loop over all the comments, looking for the first by astrometry.net<\/span>\r\n<span class=\"comment\">% which contains information about the photo.<\/span>\r\n<span class=\"keyword\">for<\/span> idx = 0:(getLength(comments)-1)\r\n    comment = comments.item(idx);\r\n    <span class=\"keyword\">if<\/span> strcmp(char(comment.getAttribute(<span class=\"string\">'authorname'<\/span>)), <span class=\"string\">'astrometry.net'<\/span>)\r\n        <span class=\"comment\">% We've found the comment, extract the comment text.<\/span>\r\n        commentText = char(comment.getTextContent());\r\n        <span class=\"comment\">% Pick out the location information into a row vector by<\/span>\r\n        <span class=\"comment\">% using a regular expression.<\/span>\r\n        locationText = regexprep(<span class=\"keyword\">...<\/span>\r\n            commentText, <span class=\"string\">'.*center: *\\(([-0-9,. ]+)\\) *degrees.*'<\/span>, <span class=\"string\">'$1'<\/span>);\r\n        location     = sscanf(locationText, <span class=\"string\">'%g,'<\/span>, [1, 2]);\r\n        <span class=\"keyword\">return<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n<span class=\"comment\">%<\/span>\r\n<span class=\"comment\">% We didn't find the astrometry.net comment, so location is unknown.<\/span>\r\nlocation = [NaN, NaN];\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><h4>Use <tt>parfeval<\/tt> to submit work to the parallel workers to read the images<a name=\"a7a8f5d5-fe62-46dc-9bae-ceb9edf41d77\"><\/a><\/h4><p>For each search result, we will make a <tt>parfeval<\/tt> request for that particular result. This will cause one of the workers in our <a href=\"https:\/\/www.mathworks.com\/help\/distcomp\/parpool.html\">parallel pool<\/a> to invoke that function. If a parallel pool is not already open, then one will be created automatically. Each <tt>parfeval<\/tt> call returns a <a title=\"https:\/\/www.mathworks.com\/help\/distcomp\/parallel.future.html (link no longer works)\"><tt>parallel.Future<\/tt><\/a> instance. The <tt>parfeval<\/tt> requests are executed in order by the parallel pool workers. The syntax of <tt>parfeval<\/tt> is rather similar to that of the MATLAB function <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/feval.html\"><tt>feval<\/tt><\/a>, except that because the evaluation doesn't take place immediately, you must specify how many output arguments you want to request. In this case, we want 3 outputs from <tt>getAstrometryPhotoFromFlickr<\/tt>.<\/p><pre class=\"codeinput\"><span class=\"keyword\">for<\/span> idx = 1:numPhotos\r\n    futures(idx) = parfeval(@getAstrometryPhotoFromFlickr, 3, <span class=\"keyword\">...<\/span>\r\n                            appKey, photoInfo{idx});\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><pre class=\"codeoutput\">Starting parallel pool (parpool) using the 'local' profile ... connected to 6 workers.\r\n<\/pre><h4>Prepare a figure for displaying the results<a name=\"aeb6e231-868b-4cc7-a046-d172e0826873\"><\/a><\/h4><p>While the workers are busy retrieving images from Flickr, we can set up a figure to display those images.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Create a new figure, initially not visible.<\/span>\r\nfigHandle = figure(<span class=\"string\">'Position'<\/span>, [200, 200, 600, 800], <span class=\"keyword\">...<\/span>\r\n                   <span class=\"string\">'Units'<\/span>, <span class=\"string\">'pixels'<\/span>, <span class=\"string\">'Visible'<\/span>, <span class=\"string\">'off'<\/span>);\r\n\r\n<span class=\"comment\">% Use FEX submission 'tight_subplot' to set up the axes<\/span>\r\n<span class=\"comment\">% https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/27991-tight-subplot-nh--nw--gap--marg-h--marg-w-<\/span>\r\naxesHandles = tight_subplot(4, 3, [0.06, 0.01], [0.01, 0.06], 0.01);\r\naxis(axesHandles, <span class=\"string\">'off'<\/span>);\r\n<\/pre><h4>Use <tt>fetchNext<\/tt> to retrieve results and then display them<a name=\"fe8c2edf-9e8d-45a1-8424-abfc8a1fd429\"><\/a><\/h4><p>We now enter a loop where we wait for results to become available on the workers. We use <a title=\"https:\/\/www.mathworks.com\/help\/distcomp\/fetchnext.html (link no longer works)\"><tt>fetchNext<\/tt><\/a> to achieve this. We pass <tt>fetchNext<\/tt> our array <tt>futures<\/tt>, and it returns when another element has completed and a new result is available. We can also specify a maximum time to wait for each new result, so that if the web service takes a really long time to run, we can abort execution gracefully.<\/p><p>Note that the results from <tt>fetchNext<\/tt> can arrive in any order, and we use the first output argument <tt>originalIdx<\/tt> to work out which element of <tt>futures<\/tt> the result corresponds to.<\/p><p>We're going to specify an upper bound on how long we're going to wait for all the futures to complete. If we exceed that time limit and there are still futures running, we can cancel them.<\/p><pre class=\"codeinput\">overallTimeLimit = 10; <span class=\"comment\">% seconds<\/span>\r\n\r\nt = tic;\r\nset(figHandle, <span class=\"string\">'Visible'<\/span>, <span class=\"string\">'on'<\/span>);\r\nnumCompleted = 0;\r\n<span class=\"keyword\">while<\/span> numCompleted &lt; numPhotos\r\n    <span class=\"keyword\">try<\/span>\r\n        <span class=\"comment\">% This blocks for up to 1 second waiting for a result.<\/span>\r\n        [originalIdx, photo, location, user] = fetchNext(futures, 1);\r\n    <span class=\"keyword\">catch<\/span> E\r\n        <span class=\"comment\">% Sometimes, the URL simply cannot be read, so an<\/span>\r\n        <span class=\"comment\">% error is thrown by the worker. Let's display that<\/span>\r\n        <span class=\"comment\">% and carry on.<\/span>\r\n        warning(<span class=\"string\">'Failed to read an image: %s'<\/span>, getReport(E));\r\n        originalIdx = [];\r\n    <span class=\"keyword\">end<\/span>\r\n\r\n    <span class=\"comment\">% If 'fetchNext' completed successfully, originalIdx<\/span>\r\n    <span class=\"comment\">% will be non-empty, and will contain the index into 'futures'<\/span>\r\n    <span class=\"comment\">% corresponding to the work that has just finished.<\/span>\r\n    <span class=\"keyword\">if<\/span> ~isempty(originalIdx)\r\n        <span class=\"comment\">% Display attribution for photo and link to the original<\/span>\r\n        fprintf(<span class=\"string\">'Photo %2d by Flickr user: %s\\n'<\/span>, originalIdx, user);\r\n        info = photoInfo{originalIdx};\r\n        fprintf(<span class=\"string\">'Original at: http:\/\/www.flickr.com\/%s\/%s\/\\n'<\/span>, info.owner, info.id);\r\n\r\n        <span class=\"comment\">% Pick our axes:<\/span>\r\n        axesToUse = axesHandles(originalIdx);\r\n\r\n        <span class=\"comment\">% Display the image<\/span>\r\n        image(photo, <span class=\"string\">'Parent'<\/span>, axesToUse);\r\n        axis(axesToUse, <span class=\"string\">'off'<\/span>); axis(axesToUse, <span class=\"string\">'equal'<\/span>);\r\n\r\n        <span class=\"comment\">% Set the title of the axis to be the location<\/span>\r\n        title(axesToUse, num2str(location));\r\n\r\n        numCompleted = numCompleted + 1;\r\n    <span class=\"keyword\">elseif<\/span> toc(t) &gt; overallTimeLimit\r\n        <span class=\"comment\">% We have exceeded our time budget!<\/span>\r\n        disp(<span class=\"string\">'Time limit expired!'<\/span>);\r\n\r\n        <span class=\"comment\">% Cancelling the futures stops execution of any running<\/span>\r\n        <span class=\"comment\">% futures, but has no effect on already-completed futures.<\/span>\r\n        cancel(futures);\r\n        <span class=\"keyword\">break<\/span>;\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\ntoc(t);\r\n<\/pre><pre class=\"codeoutput\">Photo  6 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/4145493112\/\r\nPhoto  2 by Flickr user: davedehetre\r\nOriginal at: http:\/\/www.flickr.com\/22433418@N04\/4950431138\/\r\nPhoto  5 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/8541349810\/\r\nPhoto  1 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/4553112538\/\r\nPhoto  4 by Flickr user: Torben Bj&oslash;rn Hansen\r\nOriginal at: http:\/\/www.flickr.com\/21067003@N07\/6105409913\/\r\nPhoto  3 by Flickr user: Harry Thomas Photography\r\nOriginal at: http:\/\/www.flickr.com\/84598277@N08\/8882356507\/\r\nPhoto  7 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/4144380775\/\r\nPhoto  8 by Flickr user: Jody Roberts\r\nOriginal at: http:\/\/www.flickr.com\/8374715@N06\/8719966332\/\r\nPhoto  9 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/4145140448\/\r\nPhoto 11 by Flickr user: davedehetre\r\nOriginal at: http:\/\/www.flickr.com\/22433418@N04\/4954464378\/\r\nPhoto 12 by Flickr user: s58y\r\nOriginal at: http:\/\/www.flickr.com\/45032885@N04\/4535865643\/\r\nPhoto 10 by Flickr user: cfaobam\r\nOriginal at: http:\/\/www.flickr.com\/33763963@N05\/7060054229\/\r\nElapsed time is 2.897368 seconds.\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2013\/parfevalFlickr_01.png\" alt=\"\"> <h4>Conclusions<a name=\"410d2da5-858f-4490-9051-6493a85bad97\"><\/a><\/h4><p>We have seen how using the <tt>parfeval<\/tt> command can be used to retrieve data from a web service, and display the results interactively as they become available. <tt>parfeval<\/tt> allows you more flexibility than <tt>parfor<\/tt> to:<\/p><div><ol><li>Plot results while the parallel computations proceed<\/li><li>Gracefully break out of parallel computations<\/li><\/ol><\/div><p>If you've ever wanted to show a waitbar or plot partial results during a <tt>parfor<\/tt> loop, then you might be able to adapt your code to use <tt>parfeval<\/tt>. If you have any further ideas, let us know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=818#respond\">here<\/a>.<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_d57cfe702dfd4a0fbd79d2978b5fb92e() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='d57cfe702dfd4a0fbd79d2978b5fb92e ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' d57cfe702dfd4a0fbd79d2978b5fb92e';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2013 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_d57cfe702dfd4a0fbd79d2978b5fb92e()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2013b<br><\/p><p class=\"footer\"><br>\r\n      Published with MATLAB&reg; R2013b<br><\/p><\/div><!--\r\nd57cfe702dfd4a0fbd79d2978b5fb92e ##### SOURCE BEGIN #####\r\n%% Getting Data from a Web API in Parallel\r\n% I'd like to introduce this week's guest blogger\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/answers\/contributors\/1287414-edric-ellis Edric Ellis>. \r\n% Edric works for the Parallel Computing development team here at\r\n% The MathWorks. In this post he will talk about using the |parfeval| command in\r\n% Parallel Computing Toolbox.\r\n%\r\n% In release R2013b, we introduced a new function\r\n% <https:\/\/www.mathworks.com\/help\/distcomp\/parfeval.html |parfeval|> to Parallel\r\n% Computing Toolbox which allows you to run MATLAB(R) functions on workers in a\r\n% parallel pool without blocking your desktop MATLAB. This allows you to create\r\n% parallel programs that are more loosely structured than those using either\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/parfor.html |parfor|> or\r\n% <https:\/\/www.mathworks.com\/help\/distcomp\/spmd.html |spmd|>. You can also have\r\n% your desktop MATLAB process the results as they become available, allowing you\r\n% to create more interactive parallel programs.\r\n%\r\n% We're going to take a look at using |parfeval| to speed up accessing a web\r\n% service by making multiple requests in parallel, and then displaying each\r\n% result as soon as it is available. We're going to query the\r\n% <http:\/\/www.flickr.com Flickr> image sharing site for astronomical images that\r\n% have structured data associated with them. We'll approach the problem in\r\n% three phases:\r\n%\r\n% # Query the web service for relevant images\r\n% # Use |parfeval| to submit work to the parallel workers to read the images\r\n% # Use |fetchNext| to retrieve results and then display them.\r\n%\r\n%% Query the web service for relevant images\r\n% Flickr provides a web-based API that allows you to query its large database of\r\n% images. To use the API, you submit a structured query to their servers, and an\r\n% XML document is returned to you. Retrieving the resulting image data from\r\n% Flickr takes some time, and this can run more quickly by performing the\r\n% retrievals in parallel. This is an example of an\r\n% <http:\/\/en.wikipedia.org\/wiki\/I\/O_bound \"I\/O-bound\"> operation.\r\n%\r\n% The Flickr web API is fully documented\r\n% <http:\/\/www.flickr.com\/services\/api\/ here>. To use the API, you need a Flickr\r\n% account, and you need to <http:\/\/www.flickr.com\/services\/apps\/create\/ request an 'API Key'> which is included in each request\r\n% to the API. Once you've got your own key, you can place it in a function\r\n% called |flickrApiKey| which simply returns the key as a string.\r\n\r\nappKey = flickrApiKey;\r\n\r\n%% Build the search URL and interpret the XML response\r\n% We're going to search for images which have a specific tag:\r\n% <http:\/\/www.flickr.com\/photos\/tags\/astrometrydotnet%3Astatus%3Dsolved\/\r\n% |astrometrydotnet:status=solved|>. These images are submitted to a Flickr\r\n% group, and then <http:\/\/astrometry.net\/ astrometry.net> analyses and annotates\r\n% the images. We build a search url, and then let\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/xmlread.html |xmlread|> read\r\n% the results from that URL.\r\n\r\nurl = ['http:\/\/api.flickr.com\/services\/rest\/', ...       % API Base URL\r\n       '?method=flickr.photos.search', ...               % API Method\r\n       '&api_key=', appKey, ...                          % Our API key\r\n       '&tags=astrometrydotnet%3Astatus%3Dsolved', ...   % Tag to search\r\n       '&license=4', ...                                 % Creative Commons\r\n       '&sort=interestingness-desc', ...                 % Sort order\r\n       '&per_page=12', ...                               % Limit results to 12\r\n       '&media=photos'];                                 % Only return photos\r\nresponse = xmlread(url);\r\n\r\n% Get all the 'photo' elements from the document.\r\nphotos    = response.getDocumentElement.getElementsByTagName('photo');\r\nnumPhotos = getLength(photos);\r\n\r\n% Ensure our search query return some results.\r\nif numPhotos == 0\r\n    error('Failed to retrieve photos from Flickr. XML response was: %s', ...\r\n          xmlwrite(response));\r\nend\r\n\r\n% Otherwise, convert the search results into a cell array of structures\r\n% containing the search result information.\r\nphotoInfo = cell(1, numPhotos);\r\nfor idx = 1:numPhotos\r\n    node = photos.item(idx-1);\r\n    photoInfo{idx}  = struct('farm',   char(node.getAttribute('farm')), ...\r\n                             'server', char(node.getAttribute('server')), ...\r\n                             'id',     char(node.getAttribute('id')), ...\r\n                             'secret', char(node.getAttribute('secret')), ...\r\n                             'owner',  char(node.getAttribute('owner')));\r\nend\r\n\r\n%% Define a function to retrieve and process the photos\r\n% For each search result, we need to run a function to retrieve\r\n% the image data. The function also searches through the Flickr comments for the\r\n% structured information from astrometry.net, and returns the direction\r\n% the telescope was pointing. In the next stage, we'll submit multiple\r\n% invocations of this function using |parfeval|.\r\n%\r\n%   function [photo, location, username] = getAstrometryPhotoFromFlickr(appKey, info)\r\n%   % First, build up the photo URL as per the Flickr documentation\r\n%   % here: <http:\/\/www.flickr.com\/services\/api\/misc.urls.html>\r\n%   url = sprintf('http:\/\/farm%s.staticflickr.com\/%s\/%s_%s.jpg', ...\r\n%                 info.farm, info.server, info.id, info.secret);\r\n%   try\r\n%       photo = imread(url);\r\n%   catch E\r\n%       % Sometimes the read fails. Try one more time.\r\n%       photo = imread(url);\r\n%   end\r\n%   %\r\n%   % Get the photo info to extract the username\r\n%   url = ['http:\/\/api.flickr.com\/services\/rest\/', ...\r\n%          '?method=flickr.photos.getInfo', ...\r\n%          '&api_key=', appKey, '&photo_id=', info.id];\r\n%   response = xmlread(url);\r\n%   owner = response.getDocumentElement.getElementsByTagName('owner');\r\n%   if getLength(owner) == 1\r\n%       username = char(owner.item(0).getAttribute('username'));\r\n%   else\r\n%       username = 'Unknown';\r\n%   end\r\n%   %\r\n%   % Next, look through the comments for the photo to extract\r\n%   % the annotation from astrometry.net.\r\n%   url = ['http:\/\/api.flickr.com\/services\/rest\/', ...\r\n%          '?method=flickr.photos.comments.getList', ...\r\n%          '&api_key=', appKey, '&photo_id=', info.id];\r\n%   response = xmlread(url);\r\n%   %\r\n%   % Get all the actual comment elements from the XML response.\r\n%   comments = response.getDocumentElement.getElementsByTagName('comment');\r\n%   %\r\n%   % Loop over all the comments, looking for the first by astrometry.net\r\n%   % which contains information about the photo.\r\n%   for idx = 0:(getLength(comments)-1)\r\n%       comment = comments.item(idx);\r\n%       if strcmp(char(comment.getAttribute('authorname')), 'astrometry.net')\r\n%           % We've found the comment, extract the comment text.\r\n%           commentText = char(comment.getTextContent());\r\n%           % Pick out the location information into a row vector by\r\n%           % using a regular expression.\r\n%           locationText = regexprep(...\r\n%               commentText, '.*center: *\\(([-0-9,. ]+)\\) *degrees.*', '$1');\r\n%           location     = sscanf(locationText, '%g,', [1, 2]);\r\n%           return\r\n%       end\r\n%   end\r\n%   %\r\n%   % We didn't find the astrometry.net comment, so location is unknown.\r\n%   location = [NaN, NaN];\r\n%   end\r\n\r\n%% Use |parfeval| to submit work to the parallel workers to read the images\r\n% For each search result, we will make a |parfeval| request for that particular\r\n% result. This will cause one of the workers in our\r\n% <https:\/\/www.mathworks.com\/help\/distcomp\/parpool.html parallel pool> to\r\n% invoke that function. If a parallel pool is not already open, then one will be\r\n% created automatically. Each |parfeval| call returns a\r\n% <https:\/\/www.mathworks.com\/help\/distcomp\/parallel.future.html |parallel.Future|> \r\n% instance. The |parfeval| requests are executed in order by the parallel\r\n% pool workers. The syntax of |parfeval| is rather similar to that of the MATLAB\r\n% function <https:\/\/www.mathworks.com\/help\/matlab\/ref\/feval.html |feval|>, except\r\n% that because the evaluation doesn't take place immediately, you must\r\n% specify how many output arguments you want to request. In this case,\r\n% we want 3 outputs from |getAstrometryPhotoFromFlickr|.\r\nfor idx = 1:numPhotos\r\n    futures(idx) = parfeval(@getAstrometryPhotoFromFlickr, 3, ...\r\n                            appKey, photoInfo{idx});\r\nend\r\n\r\n%% Prepare a figure for displaying the results\r\n% While the workers are busy retrieving images from Flickr, we\r\n% can set up a figure to display those images.\r\n\r\n% Create a new figure, initially not visible.\r\nfigHandle = figure('Position', [200, 200, 600, 800], ...\r\n                   'Units', 'pixels', 'Visible', 'off');\r\n\r\n% Use FEX submission 'tight_subplot' to set up the axes\r\n% https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/27991-tight-subplot-nh--nw--gap--marg-h--marg-w-\r\naxesHandles = tight_subplot(4, 3, [0.06, 0.01], [0.01, 0.06], 0.01);\r\naxis(axesHandles, 'off');\r\n\r\n%% Use |fetchNext| to retrieve results and then display them\r\n% We now enter a loop where we wait for results to become available on\r\n% the workers. We use <https:\/\/www.mathworks.com\/help\/distcomp\/fetchnext.html |fetchNext|>\r\n% to achieve this. We pass |fetchNext| our array |futures|, and it\r\n% returns when another element has completed and a new result\r\n% is available. We can also specify a maximum time to wait for\r\n% each new result, so that if the web service takes a really\r\n% long time to run, we can abort execution gracefully.\r\n%\r\n% Note that the results from |fetchNext| can arrive in any order, and we use the\r\n% first output argument |originalIdx| to work out which element of |futures| the\r\n% result corresponds to.\r\n%\r\n% We're going to specify an upper bound on how long we're going to wait for all\r\n% the futures to complete. If we exceed that time limit and there are still\r\n% futures running, we can cancel them.\r\n\r\noverallTimeLimit = 10; % seconds\r\n\r\nt = tic;\r\nset(figHandle, 'Visible', 'on');\r\nnumCompleted = 0;\r\nwhile numCompleted < numPhotos\r\n    try\r\n        % This blocks for up to 1 second waiting for a result.\r\n        [originalIdx, photo, location, user] = fetchNext(futures, 1);\r\n    catch E\r\n        % Sometimes, the URL simply cannot be read, so an\r\n        % error is thrown by the worker. Let's display that\r\n        % and carry on.\r\n        warning('Failed to read an image: %s', getReport(E));\r\n        originalIdx = [];\r\n    end\r\n\r\n    % If 'fetchNext' completed successfully, originalIdx\r\n    % will be non-empty, and will contain the index into 'futures'\r\n    % corresponding to the work that has just finished.\r\n    if ~isempty(originalIdx)\r\n        % Display attribution for photo and link to the original\r\n        fprintf('Photo %2d by Flickr user: %s\\n', originalIdx, user);\r\n        info = photoInfo{originalIdx};\r\n        fprintf('Original at: http:\/\/www.flickr.com\/%s\/%s\/\\n', info.owner, info.id);\r\n\r\n        % Pick our axes:\r\n        axesToUse = axesHandles(originalIdx);\r\n\r\n        % Display the image\r\n        image(photo, 'Parent', axesToUse);\r\n        axis(axesToUse, 'off'); axis(axesToUse, 'equal');\r\n\r\n        % Set the title of the axis to be the location\r\n        title(axesToUse, num2str(location));\r\n\r\n        numCompleted = numCompleted + 1;\r\n    elseif toc(t) > overallTimeLimit\r\n        % We have exceeded our time budget!\r\n        disp('Time limit expired!');\r\n\r\n        % Cancelling the futures stops execution of any running\r\n        % futures, but has no effect on already-completed futures.\r\n        cancel(futures);\r\n        break;\r\n    end\r\nend\r\ntoc(t);\r\n\r\n%% Conclusions\r\n% We have seen how using the |parfeval| command can be used to retrieve data\r\n% from a web service, and display the results interactively as they become\r\n% available. |parfeval| allows you more flexibility than |parfor| to:\r\n%\r\n% # Plot results while the parallel computations proceed\r\n% # Gracefully break out of parallel computations\r\n%\r\n% If you've ever wanted to show a waitbar or plot partial results during a\r\n% |parfor| loop, then you might be able to adapt your code to use |parfeval|. If\r\n% you have any further ideas, let us know <https:\/\/blogs.mathworks.com\/loren\/?p=818#respond here>.\r\n\r\n\r\n##### SOURCE END ##### d57cfe702dfd4a0fbd79d2978b5fb92e\r\n-->\r\n","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2013\/parfevalFlickr_01.png\" onError=\"this.style.display ='none';\" \/><\/div><!--introduction--><p>I'd like to introduce this week's guest blogger <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/answers\/contributors\/1287414-edric-ellis\">Edric Ellis<\/a>. Edric works for the Parallel Computing development team here at The MathWorks. In this post he will talk about using the <tt>parfeval<\/tt> command in Parallel Computing Toolbox.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2013\/12\/09\/getting-data-from-a-web-api-in-parallel\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[6,34],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/818"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=818"}],"version-history":[{"count":5,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/818\/revisions"}],"predecessor-version":[{"id":823,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/818\/revisions\/823"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=818"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=818"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=818"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}