{"id":871,"date":"2017-02-07T08:44:46","date_gmt":"2017-02-07T08:44:46","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=871"},"modified":"2017-02-06T21:46:34","modified_gmt":"2017-02-06T21:46:34","slug":"connect-to-twitter-with-oauth-over-http-interface","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2017\/02\/07\/connect-to-twitter-with-oauth-over-http-interface\/","title":{"rendered":"Accessing Twitter with HTTP Interface in MATLAB"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p><i>Folks, today we have a nice collaboration between blogs. If you haven't already seen it, <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/951521\">Toshi Takeuchi<\/a> just posted on Loren's blog today to help you <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=2209\">analyze fake news<\/a>. However, he is also here today to teach you how to fish, so to speak, so you are empowered to use the protocol to transfer hypertext all over the interweb! Here's Toshi...<\/i><\/p><p>When it comes to Twitter, our usual go-to tool is venerable <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/34837-twitty\">Twitty<\/a> by <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/1421693-vladimir-bondarenko\">Vladimir Bondarenko<\/a>. Twitty hasn't been updated since July 2013 and you may want to take advantage of newer features in MATLAB, such as <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/string.html\">string arrays<\/a> and <tt><a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/jsonencode.html\">jsonencode<\/a><\/tt> and <tt><a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/jsondecode.html\">jsondecode<\/a><\/tt> functions.<\/p><p>One of the new exciting features introduced in R2016b is <a title=\"https:\/\/www.mathworks.com\/help\/matlab\/call-wsdl-web-services_bu9hx2b-1.html (link no longer works)\">HTTP Interface<\/a>, and we can use it to accessTwitter API and deal with its <a title=\"https:\/\/dev.twitter.com\/oauth\/overview (link no longer works)\">OAuth requirements<\/a> without Twitty if you want to roll your own tool. Let's search for tweets related to NASA as an example.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/www.mathworks.com\/content\/mathworks\/www\/en\/company\/user_stories\/trivector-verifies-time-latencies-for-ares-i-rocket\/jcr:content\/imageEnhancedParsys\/image.adapt.full.high.jpg\/1469941207398.jpg\" alt=\"\"> <\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#3f147561-7bd9-455b-a6a7-c8e59edcddf3\">Twitty as reference<\/a><\/li><li><a href=\"#692a4ac4-5593-4eea-9b9a-2820654b1a8d\">Twitter Credentials<\/a><\/li><li><a href=\"#e0d53f65-63f0-4e31-801e-933cfcb330a4\">Percent Encoding<\/a><\/li><li><a href=\"#193e6266-2030-4dac-9d27-8a4bbf64e9ef\">Twitter Search API<\/a><\/li><li><a href=\"#3717eaf8-57f3-4f9c-97a4-a5fbe441c7e2\">Set up OAuth Parameters for Authentication<\/a><\/li><li><a href=\"#f9a07aeb-4a8e-42a4-a73b-2e0265abed5d\">Generate Signature with HMAC-SH1 Encryption<\/a><\/li><li><a href=\"#e401cb5c-f3f9-4e1f-be7e-be0cbbddbf17\">Build the OAuth Header<\/a><\/li><li><a href=\"#905a92bc-ded9-4fcc-817b-8f7c5a5c8777\">Send HTTP Request<\/a><\/li><li><a href=\"#76c7e4f4-cc78-48d5-918a-316ee37ab787\">Process Response<\/a><\/li><li><a href=\"#3b889a0f-3167-4b71-8c82-2d79d201e101\">Summary<\/a><\/li><\/ul><\/div><h4>Twitty as reference<a name=\"3f147561-7bd9-455b-a6a7-c8e59edcddf3\"><\/a><\/h4><p>Twitty is still a good reference for how we deal with <a href=\"https:\/\/en.wikipedia.org\/wiki\/OAuth\">OAuth authentication<\/a>, which is taken care of in  <tt>callTwitterAPI<\/tt> method. I would like to walk through this method, replacing Java calls with HTTP Interface where applicable. To follow along please also check <a href=\"https:\/\/blogs.mathworks.com\/developer\/files\/oauth.pdf\">this diagram<\/a>.<\/p><pre class=\"codeinput\">dbtype <span class=\"string\">twitty_1.1.1\/twitty.m<\/span> <span class=\"string\">1816:1819<\/span>\r\n<\/pre><pre class=\"codeoutput\">\r\n1816  %% The main API caller\r\n1817      methods(Access = private)\r\n1818          function S = callTwitterAPI(twtr,httpMethod,url,params,~)\r\n1819          % Call to the twitter API. \r\n<\/pre><h4>Twitter Credentials<a name=\"692a4ac4-5593-4eea-9b9a-2820654b1a8d\"><\/a><\/h4><p>I am assuming that you already have <a href=\"https:\/\/developer.twitter.com\/en\/docs\/basics\/authentication\/guides\/access-tokens\">your developer credentials<\/a> and downloaded Twitty into your curent folder. The workspace variable <tt>creds<\/tt> should contain your credentials in a struct in the following format:<\/p><pre class=\"codeinput\">creds = struct;                                             <span class=\"comment\">% example<\/span>\r\ncreds.ConsumerKey = <span class=\"string\">'your consumer key'<\/span>;\r\ncreds.ConsumerSecret = <span class=\"string\">'your consumer secret'<\/span>;\r\ncreds.AccessToken = <span class=\"string\">'your token'<\/span>;\r\ncreds.AccessTokenSecret = <span class=\"string\">'your token secret'<\/span>;\r\nload <span class=\"string\">creds<\/span>                                                  <span class=\"comment\">% load my real credentials<\/span>\r\n<\/pre><h4>Percent Encoding<a name=\"e0d53f65-63f0-4e31-801e-933cfcb330a4\"><\/a><\/h4><p>The <a title=\"https:\/\/dev.twitter.com\/oauth\/overview (link no longer works)\">Twitter documentation<\/a> says it is important to use proper percent encoding comforming to* *<a href=\"http:\/\/tools.ietf.org\/html\/rfc3986#section-2.1\">RFC 3986, Section 2.1<\/a>. Java's <tt><a href=\"http:\/\/docs.oracle.com\/javase\/1.5.0\/docs\/api\/java\/net\/URLEncoder.html#encode%28java.lang.String,%20java.lang.String%29\">URLEncoder.encode<\/a><\/tt> does almost everything except for a space, which becomes <tt>'+'<\/tt> rather than <tt>'%20'<\/tt>. Let's fix that with an anonymous function <tt>pctencode<\/tt>. Twitty also does the same.<\/p><pre class=\"codeinput\">pctencode = @(str) <span class=\"keyword\">...<\/span><span class=\"comment\">                                      % anonymous function<\/span>\r\n    replace(char(java.net.URLEncoder.encode(str,<span class=\"string\">'UTF-8'<\/span>) ),<span class=\"string\">'+'<\/span>,<span class=\"string\">'%20'<\/span>);\r\nquery = <span class=\"string\">'matlab http interface (new in R2016b)'<\/span>;            <span class=\"comment\">% sample text to encode<\/span>\r\ndisp(pctencode(query))                                      <span class=\"comment\">% encoded text<\/span>\r\n<\/pre><pre class=\"codeoutput\">matlab%20http%20interface%20%28new%20in%20R2016b%29\r\n<\/pre><h4>Twitter Search API<a name=\"193e6266-2030-4dac-9d27-8a4bbf64e9ef\"><\/a><\/h4><p>Now let's try the <a href=\"https:\/\/dev.twitter.com\/rest\/public\/search\">Twitter Search API<\/a> to search for tweets about NASA. I am also going to use string arrays to vectorize string handling.<\/p><pre class=\"codeinput\">base_url = <span class=\"string\">'https:\/\/api.twitter.com\/1.1\/search\/tweets.json'<\/span>;<span class=\"comment\">% search API URL<\/span>\r\nparams = struct;                                            <span class=\"comment\">% initialize params<\/span>\r\nparams.q = pctencode(<span class=\"string\">'nasa'<\/span>);                               <span class=\"comment\">% search term<\/span>\r\nparams.include_entities = urlencode(<span class=\"string\">'true'<\/span>);                <span class=\"comment\">% search param<\/span>\r\nkeys = string(fieldnames(params));                          <span class=\"comment\">% get field names<\/span>\r\nvalues = string(struct2cell(params));                       <span class=\"comment\">% get field values<\/span>\r\n\r\n<span class=\"comment\">% vectorized string handling<\/span>\r\nkeys = keys + <span class=\"string\">'='<\/span>;                                          <span class=\"comment\">% add =<\/span>\r\nvalues = values + <span class=\"string\">'&amp;'<\/span>;                                      <span class=\"comment\">% add &amp;<\/span>\r\nqueryStr = join([keys, values],<span class=\"string\">''<\/span>);                         <span class=\"comment\">% join columns<\/span>\r\nqueryStr = join(queryStr(:),<span class=\"string\">''<\/span>);                            <span class=\"comment\">% convert to a single string<\/span>\r\nqueryStr{1}(end) = [];                                      <span class=\"comment\">% remove extra &amp;<\/span>\r\ndisp(queryStr)                                              <span class=\"comment\">% print string<\/span>\r\n<\/pre><pre class=\"codeoutput\">q=nasa&amp;include_entities=true\r\n<\/pre><h4>Set up OAuth Parameters for Authentication<a name=\"3717eaf8-57f3-4f9c-97a4-a5fbe441c7e2\"><\/a><\/h4><p><a title=\"https:\/\/dev.twitter.com\/oauth\/overview (link no longer works)\">Twitter documentation<\/a> also covers number of other requirements in order to implement OAuth authentication properly.<\/p><div><ul><li>Declare OAuth version<\/li><\/ul><\/div><pre class=\"codeinput\">params.oauth_version = <span class=\"string\">'1.0'<\/span>;\r\n<\/pre><div><ul><li>Create unique token for an HTTP request by generating a random alphanumeric sequence.<\/li><\/ul><\/div><pre class=\"codeinput\">params.oauth_nonce = replace([num2str(now) num2str(rand)], <span class=\"string\">'.'<\/span>, <span class=\"string\">''<\/span>);\r\n<\/pre><div><ul><li>Create a timestamp in number of seconds since the Unix epoch using <a title=\"https:\/\/www.mathworks.com\/help\/matlab\/create-date-and-time-arrays.html (link no longer works)\">datetime arrays<\/a>.<\/li><\/ul><\/div><pre class=\"codeinput\">epoch = datetime(<span class=\"string\">'1970-01-01T00:00:00Z'<\/span>,<span class=\"string\">'InputFormat'<\/span>, <span class=\"keyword\">...<\/span>\r\n    <span class=\"string\">'yyyy-MM-dd''T''HH:mm:ssXXX'<\/span>,<span class=\"string\">'TimeZone'<\/span>,<span class=\"string\">'UTC'<\/span>);\r\ncurr = datetime(<span class=\"string\">'now'<\/span>,<span class=\"string\">'TimeZone'<\/span>,<span class=\"string\">'UTC'<\/span>,<span class=\"string\">'Format'<\/span>, <span class=\"keyword\">...<\/span>\r\n    <span class=\"string\">'yyyy-MM-dd''T''HH:mm:ssXXX'<\/span>);\r\nparams.oauth_timestamp = int2str(seconds(curr - epoch));\r\n<\/pre><div><ul><li>Get user credentials.<\/li><\/ul><\/div><pre class=\"codeinput\">params.oauth_consumer_key = creds.ConsumerKey;\r\nparams.oauth_token = creds.AccessToken;\r\n<\/pre><div><ul><li>Declare HTTP request signature encryption scheme, which must be HMAC-SHA1.<\/li><\/ul><\/div><pre class=\"codeinput\">sigMethod = <span class=\"string\">'HMAC-SHA1'<\/span>;\r\nparams.oauth_signature_method = sigMethod;\r\n<\/pre><p>Now we can put it all together using vectorized string handling.<\/p><pre class=\"codeinput\">params = orderfields(params);                               <span class=\"comment\">% sort the field order<\/span>\r\nkeys = string(fieldnames(params));                          <span class=\"comment\">% get field names<\/span>\r\nvalues = string(struct2cell(params));                       <span class=\"comment\">% get field values<\/span>\r\nkeys = keys + <span class=\"string\">'='<\/span>;                                          <span class=\"comment\">% add =<\/span>\r\nvalues = values + <span class=\"string\">'&amp;'<\/span>;                                      <span class=\"comment\">% add &amp;<\/span>\r\noauthStr = join([keys, values],<span class=\"string\">''<\/span>);                         <span class=\"comment\">% join columns<\/span>\r\noauthStr = join(oauthStr(:),<span class=\"string\">''<\/span>);                            <span class=\"comment\">% convert to a single string<\/span>\r\noauthStr{1}(end) = [];                                      <span class=\"comment\">% remove extra &amp;<\/span>\r\n<\/pre><h4>Generate Signature with HMAC-SH1 Encryption<a name=\"f9a07aeb-4a8e-42a4-a73b-2e0265abed5d\"><\/a><\/h4><p>Create the signature base string and signature key.<\/p><pre class=\"codeinput\">httpMethod = string(<span class=\"string\">'GET'<\/span>);                                 <span class=\"comment\">% set http method to get<\/span>\r\nsigStr = upper(httpMethod);                                 <span class=\"comment\">% add it to sigStr<\/span>\r\nsigStr = [sigStr pctencode(base_url)];                      <span class=\"comment\">% add base Url<\/span>\r\nsigStr = [sigStr pctencode(char(oauthStr))];                <span class=\"comment\">% add oauthStr<\/span>\r\nsigStr = char(join(sigStr,<span class=\"string\">'&amp;'<\/span>));                            <span class=\"comment\">% join them with &amp;<\/span>\r\nsigKey = [creds.ConsumerSecret <span class=\"string\">'&amp;'<\/span> creds.AccessTokenSecret];<span class=\"comment\">% create sigKey<\/span>\r\n<\/pre><p>Generate the signature with HMAC-SHA1 using Java as Twitty does.<\/p><pre class=\"codeinput\">import <span class=\"string\">javax.crypto.Mac<\/span>                                     <span class=\"comment\">% import Java libraries<\/span>\r\nimport <span class=\"string\">javax.crypto.spec.SecretKeySpec<\/span>;                     <span class=\"comment\">% key spec methods<\/span>\r\nimport <span class=\"string\">org.apache.commons.codec.binary.Base64<\/span>               <span class=\"comment\">% base64 codec<\/span>\r\n\r\nalgorithm = strrep(sigMethod,<span class=\"string\">'-'<\/span>,<span class=\"string\">''<\/span>);                       <span class=\"comment\">% use HMAC-SHA1<\/span>\r\nkey = SecretKeySpec(int8(sigKey), algorithm);               <span class=\"comment\">% get key<\/span>\r\nmac = Mac.getInstance(algorithm);                           <span class=\"comment\">% message auth code<\/span>\r\nmac.init(key);                                              <span class=\"comment\">% initialize mac<\/span>\r\nmac.update(int8(sigStr));                                   <span class=\"comment\">% add base string<\/span>\r\nparams.oauth_signature = <span class=\"keyword\">...<\/span><span class=\"comment\">                                % get signature<\/span>\r\n    char(Base64.encodeBase64(mac.doFinal)');\r\nparams = orderfields(params);                               <span class=\"comment\">% order fields<\/span>\r\n<\/pre><h4>Build the OAuth Header<a name=\"e401cb5c-f3f9-4e1f-be7e-be0cbbddbf17\"><\/a><\/h4><p>With the signature added to the parameter, we can now build the OAuth header.<\/p><pre class=\"codeinput\">keys = fieldnames(params);                                  <span class=\"comment\">% get field names<\/span>\r\nisOauth = contains(keys,<span class=\"string\">'oauth'<\/span>);                           <span class=\"comment\">% oauth params<\/span>\r\nkeys = cellfun(pctencode,keys,<span class=\"string\">'UniformOutput'<\/span>,false);       <span class=\"comment\">% percent-encode keys<\/span>\r\nkeys = string(keys) + <span class=\"string\">'=\"'<\/span>;                                 <span class=\"comment\">% convert to string and add =\"<\/span>\r\nvalues = struct2cell(params);                               <span class=\"comment\">% get field values<\/span>\r\nvalues = cellfun(pctencode,values,<span class=\"string\">'UniformOutput'<\/span>,false);   <span class=\"comment\">% percent-encode values<\/span>\r\nvalues = string(values) +<span class=\"string\">'\", '<\/span>;                             <span class=\"comment\">% convert to string and add \",<\/span>\r\noauth_header = join([keys, values],<span class=\"string\">''<\/span>);                     <span class=\"comment\">% join columns<\/span>\r\noauth_header = oauth_header(isOauth,:);                     <span class=\"comment\">% keep OAuth params only<\/span>\r\noauth_header = <span class=\"string\">'OAuth '<\/span> + join(oauth_header(:),<span class=\"string\">''<\/span>);         <span class=\"comment\">% convert to a single string<\/span>\r\noauth_header{1}(end-1:end) = [];                            <span class=\"comment\">% remove extra \"<\/span>\r\n<\/pre><h4>Send HTTP Request<a name=\"905a92bc-ded9-4fcc-817b-8f7c5a5c8777\"><\/a><\/h4><p>This is where everything we have done so far comes together and you get to see how you can use HTTP Interface functions to send an HTTP request, such as <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.uri-class.html\">matlab.net.URI<\/a>, <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.http.requestmethod-class.html\">matlab.net.http.RequestMethod<\/a>, <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.http.field.authenticatefield-class.html\">matlab.net.http.field.AuthenticateField<\/a>, etc. I will just load the previously saved response object here for reproducibility.<\/p><pre class=\"codeinput\">import <span class=\"string\">matlab.net.URI<\/span>;                                      <span class=\"comment\">% shorten lib calls<\/span>\r\nimport <span class=\"string\">matlab.net.http.RequestMethod<\/span>;\r\nimport <span class=\"string\">matlab.net.http.RequestMessage<\/span>;\r\nimport <span class=\"string\">matlab.net.http.field.AuthorizationField<\/span>;\r\nimport <span class=\"string\">matlab.net.http.field.ContentTypeField<\/span>;\r\n\r\nconnURI = URI(base_url,queryStr);                           <span class=\"comment\">% create URI obj<\/span>\r\nmethod = RequestMethod.(char(httpMethod));                  <span class=\"comment\">% create method obj<\/span>\r\nauth = AuthorizationField(<span class=\"string\">'Authorization'<\/span>,oauth_header);    <span class=\"comment\">% create auth field obj<\/span>\r\nct_value = <span class=\"string\">'application\/x-www-form-urlencoded'<\/span>;             <span class=\"comment\">% content type<\/span>\r\ncontent_type = ContentTypeField(ct_value);                  <span class=\"comment\">% create content type obj<\/span>\r\nheader = [auth content_type];                               <span class=\"comment\">% combine header fields objs<\/span>\r\nrequest = RequestMessage(method,header);                    <span class=\"comment\">% create request obj<\/span>\r\nresponse = send(request,connURI);                           <span class=\"comment\">% send http request<\/span>\r\nload <span class=\"string\">response<\/span>                                               <span class=\"comment\">% load saved resp<\/span>\r\ndisp(response)\r\n<\/pre><pre class=\"codeoutput\">  ResponseMessage with properties:\r\n\r\n    StatusLine: ' 200 OK'\r\n    StatusCode: OK\r\n        Header: [1&times;25 matlab.net.http.HeaderField]\r\n          Body: [1&times;1 matlab.net.http.MessageBody]\r\n     Completed: 0\r\n\r\n<\/pre><h4>Process Response<a name=\"76c7e4f4-cc78-48d5-918a-316ee37ab787\"><\/a><\/h4><p>The Twitter API returns its response in JSON format, but the HTTP interface in MATLAB automatically parses it into a structure array. Now you can process it with any familiar MATLAB functions of your choice.<\/p><pre class=\"codeinput\"><span class=\"keyword\">if<\/span> string(fieldnames(response.Body.Data)) ~= <span class=\"string\">'errors'<\/span>       <span class=\"comment\">% if not errors<\/span>\r\n    s = response.Body.Data.statuses;                        <span class=\"comment\">% get tweets<\/span>\r\n    <span class=\"keyword\">if<\/span> ~isempty(s)                                          <span class=\"comment\">% if not empty<\/span>\r\n        tweets = cellfun(@(x) x.text, s, <span class=\"string\">'UniformOutput'<\/span>,0);<span class=\"comment\">% extract text<\/span>\r\n        tweets = char(replace(tweets,char(10),<span class=\"string\">' '<\/span>));        <span class=\"comment\">% remove new line<\/span>\r\n        tweets = string(tweets(:,1:70)) + <span class=\"string\">'...'<\/span>;            <span class=\"comment\">% truncate tweets<\/span>\r\n        disp(tweets)\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><pre class=\"codeoutput\">    \"RT @RogueNASA: 10 days, 11 hours remaining. We've almost made it to $1...\"\r\n    \"RT @gp_pulipaka: NASA Tests AI For Autonomous Exploration. #BigData #D...\"\r\n    \"RT @BillP____: - &#928;&#959;&#965; &#960;&#945;&#962;, &#928;&#945;&#953;&#948;&#943; &#956;&#959;&#965; ; - &#931;&#964;&#959; &#916;&#953;&#940;&#963;&#964;&#951;&#956;&#945; &#956;&#949; &#964;&#959;&#957; &#913;&#955;&#941;&#958;&#951;, &#924;&#940;&#957;...\"\r\n    \"Camp Century: Put on Ice, But Only for So Long : Image of the Day http...\"\r\n    \"RT @SAI: This super powerful telescope is looking at the first black h...\"\r\n    \"RT @medsymeds: Minsan talaga mahirap intindihin ang mga magulang kahit...\"\r\n    \"RT @hot_jmea: Haayy. naku Yuan...tlaga nga nmang nasa huli ang pagsisi...\"\r\n    \"RT @WIRED: &#127909;  The traditional, white NASA space suit just got a makeo...\"\r\n    \"RT @ariscfu: &#925;&#940;&#964;&#959;&#962; &#954;&#945;&#953; &#959; &#914;&#941;&#961;&#957;&#949;&#961; &#966;&#959;&#957; &#924;&#960;&#961;&#940;&#959;&#965;&#957;... #greek_NASA https:\/\/t.c...\"\r\n    \"RT @LitsaXarou: &#916;&#965;&#959; &#960;&#961;&#940;&#947;&#956;&#945;&#964;&#945; &#948;&#949;&#957; &#964;&#972;&#955;&#956;&#951;&#963;&#949; &#957;&#945; &#954;&#945;&#957;&#949;&#953; &#954;&#945;&#957;&#949;&#953; &#959; &#915;&#953;&#974;&#961;&#947;&#959;&#962; &#928;&#945;&#960;&#945;...\"\r\n    \"Simula nung jan. 26 nung nalaman ko, lagi nasa isip ko \"MAG IPON\" gust...\"\r\n    \"RT @Rika_Skafida: &#908;&#960;&#959;&#965; &#945;&#949;&#961;&#959;&#960;&#955;&#940;&#957;&#959;, &#946;&#940;&#950;&#959;&#965;&#956;&#949; &#948;&#953;&#945;&#963;&#964;&#951;&#956;&#972;&#960;&#955;&#959;&#953;&#959; #&#916;&#953;&#945;&#963;&#964;&#951;&#956;&#953;&#954;&#945;_&#965;&#960;...\"\r\n    \"RT @Fezaari12: You Only Live Once ang sagot sa tanong nasa tv screen #...\"\r\n    \"@mik3cap NASA's CI Newsletter.                                        ...\"\r\n    \"RT @vasilisxatzis13: &#924;&#949;&#964;&#940; &#945;&#960;&#972; &#964;&#953;&#962; &#960;&#949;&#961;&#953;&#954;&#959;&#960;&#941;&#962; &#963;&#964;&#951;&#957; #greek_NASA https:\/\/t...\"\r\n\r\n<\/pre><h4>Summary<a name=\"3b889a0f-3167-4b71-8c82-2d79d201e101\"><\/a><\/h4><p>In this post you saw how you can use HTTP Interface to access Twitter and you now have a foundation to build your own tool to analyze Twitter feed. HTTP interface feature offers other possibilities as well. For example, I used it to get expanded urls from shortened urls. How would you use this in your projects? Please share your ideas <a href=\"https:\/\/blogs.mathworks.com\/developer\/?p=871#respond\">here<\/a>.<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_615df350a3df4f08a3fc066c27863844() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='615df350a3df4f08a3fc066c27863844 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 615df350a3df4f08a3fc066c27863844';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2017 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_615df350a3df4f08a3fc066c27863844()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2016b<br><\/p><\/div><!--\r\n615df350a3df4f08a3fc066c27863844 ##### SOURCE BEGIN #####\r\n%% \r\n% _Folks, today we have a nice collaboration between blogs. If you haven't\r\n% already seen it,\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/951521 Toshi\r\n% Takeuchi> just posted on Loren's blog today to help you\r\n% <https:\/\/blogs.mathworks.com\/loren\/?p=2209 analyze fake news>. However, he\r\n% is also here today to teach you how to fish, so to speak, so you are\r\n% empowered to use the protocol to transfer hypertext all over the\r\n% interweb! Here's Toshi..._\r\n%\r\n% When it comes to Twitter, our usual go-to tool is venerable\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/34837-twitty\r\n% Twitty> by\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/1421693-vladimir-bondarenko\r\n% Vladimir Bondarenko>. Twitty hasn't been updated since July 2013 and you may\r\n% want to take advantage of newer features in MATLAB, such as\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/string.html string arrays> and\r\n% |<https:\/\/www.mathworks.com\/help\/matlab\/ref\/jsonencode.html jsonencode>|\r\n% and |<https:\/\/www.mathworks.com\/help\/matlab\/ref\/jsondecode.html\r\n% jsondecode>| functions. \r\n% \r\n% One of the new exciting features introduced in R2016b is\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/call-wsdl-web-services_bu9hx2b-1.html\r\n% HTTP Interface>, and we can use it to accessTwitter API and deal with its\r\n% <https:\/\/dev.twitter.com\/oauth\/overview OAuth requirements> without\r\n% Twitty if you want to roll your own tool. Let's search for tweets related\r\n% to NASA as an example.\r\n% \r\n% <<https:\/\/www.mathworks.com\/content\/mathworks\/www\/en\/company\/user_stories\/trivector-verifies-time-latencies-for-ares-i-rocket\/jcr:content\/imageEnhancedParsys\/image.adapt.full.high.jpg\/1469941207398.jpg>>\r\n\r\n%% Twitty as reference\r\n% Twitty is still a good reference for how we deal with\r\n% <https:\/\/en.wikipedia.org\/wiki\/OAuth OAuth authentication>, which is\r\n% taken care of in  |callTwitterAPI| method. I would like to walk through\r\n% this method, replacing Java calls with HTTP Interface where applicable.\r\n% To follow along please also check\r\n% <https:\/\/blogs.mathworks.com\/developer\/files\/oauth.pdf this diagram>.\r\n\r\ndbtype twitty_1.1.1\/twitty.m 1816:1819\r\n%% Twitter Credentials\r\n% I am assuming that you already have\r\n% <https:\/\/developer.twitter.com\/en\/docs\/basics\/authentication\/guides\/access-tokens your developer\r\n% credentials> and downloaded Twitty into your curent folder. The workspace\r\n% variable |creds| should contain your credentials in a struct in the\r\n% following format:\r\n\r\ncreds = struct;                                             % example\r\ncreds.ConsumerKey = 'your consumer key';\r\ncreds.ConsumerSecret = 'your consumer secret';\r\ncreds.AccessToken = 'your token';\r\ncreds.AccessTokenSecret = 'your token secret';\r\nload creds                                                  % load my real credentials\r\n\r\n%% Percent Encoding\r\n% The <https:\/\/dev.twitter.com\/oauth\/overview Twitter documentation> says\r\n% it is important to use proper percent encoding comforming to*\r\n% *<http:\/\/tools.ietf.org\/html\/rfc3986#section-2.1 RFC 3986, Section 2.1>.\r\n% Java's\r\n% |<http:\/\/docs.oracle.com\/javase\/1.5.0\/docs\/api\/java\/net\/URLEncoder.html#encode%28java.lang.String,%20java.lang.String%29\r\n% URLEncoder.encode>| does almost everything except for a space, which\r\n% becomes |'+'| rather than |'%20'|. Let's fix that with an anonymous\r\n% function |pctencode|. Twitty also does the same.\r\n\r\npctencode = @(str) ...                                      % anonymous function\r\n    replace(char(java.net.URLEncoder.encode(str,'UTF-8') ),'+','%20');\r\nquery = 'matlab http interface (new in R2016b)';            % sample text to encode        \r\ndisp(pctencode(query))                                      % encoded text\r\n\r\n%% Twitter Search API\r\n% Now let's try the <https:\/\/dev.twitter.com\/rest\/public\/search Twitter\r\n% Search API> to search for tweets about NASA. I am also going to use\r\n% string arrays to vectorize string handling.\r\n\r\nbase_url = 'https:\/\/api.twitter.com\/1.1\/search\/tweets.json';% search API URL\r\nparams = struct;                                            % initialize params\r\nparams.q = pctencode('nasa');                               % search term\r\nparams.include_entities = urlencode('true');                % search param\r\nkeys = string(fieldnames(params));                          % get field names\r\nvalues = string(struct2cell(params));                       % get field values\r\n\r\n% vectorized string handling\r\nkeys = keys + '=';                                          % add =\r\nvalues = values + '&';                                      % add & \r\nqueryStr = join([keys, values],'');                         % join columns\r\nqueryStr = join(queryStr(:),'');                            % convert to a single string\r\nqueryStr{1}(end) = [];                                      % remove extra &\r\ndisp(queryStr)                                              % print string\r\n\r\n%% Set up OAuth Parameters for Authentication\r\n% <https:\/\/dev.twitter.com\/oauth\/overview Twitter documentation> also\r\n% covers number of other requirements in order to implement OAuth\r\n% authentication properly.\r\n% \r\n% * Declare OAuth version\r\n\r\nparams.oauth_version = '1.0';\r\n%% \r\n% * Create unique token for an HTTP request by generating a random\r\n% alphanumeric sequence.\r\n\r\nparams.oauth_nonce = replace([num2str(now) num2str(rand)], '.', '');\r\n%% \r\n% * Create a timestamp in number of seconds since the Unix epoch using\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/create-date-and-time-arrays.html\r\n% datetime arrays>.\r\n\r\nepoch = datetime('1970-01-01T00:00:00Z','InputFormat', ...\r\n    'yyyy-MM-dd''T''HH:mm:ssXXX','TimeZone','UTC');\r\ncurr = datetime('now','TimeZone','UTC','Format', ...\r\n    'yyyy-MM-dd''T''HH:mm:ssXXX');\r\nparams.oauth_timestamp = int2str(seconds(curr - epoch));\r\n%% \r\n% * Get user credentials.\r\n\r\nparams.oauth_consumer_key = creds.ConsumerKey;\r\nparams.oauth_token = creds.AccessToken;\r\n%% \r\n% * Declare HTTP request signature encryption scheme, which must be\r\n% HMAC-SHA1.\r\n\r\nsigMethod = 'HMAC-SHA1';\r\nparams.oauth_signature_method = sigMethod;\r\n%% \r\n% Now we can put it all together using vectorized string handling. \r\n\r\nparams = orderfields(params);                               % sort the field order\r\nkeys = string(fieldnames(params));                          % get field names\r\nvalues = string(struct2cell(params));                       % get field values\r\nkeys = keys + '=';                                          % add =\r\nvalues = values + '&';                                      % add & \r\noauthStr = join([keys, values],'');                         % join columns\r\noauthStr = join(oauthStr(:),'');                            % convert to a single string\r\noauthStr{1}(end) = [];                                      % remove extra &\r\n\r\n%% Generate Signature with HMAC-SH1 Encryption\r\n% Create the signature base string and signature key.\r\n\r\nhttpMethod = string('GET');                                 % set http method to get\r\nsigStr = upper(httpMethod);                                 % add it to sigStr\r\nsigStr = [sigStr pctencode(base_url)];                      % add base Url\r\nsigStr = [sigStr pctencode(char(oauthStr))];                % add oauthStr\r\nsigStr = char(join(sigStr,'&'));                            % join them with &\r\nsigKey = [creds.ConsumerSecret '&' creds.AccessTokenSecret];% create sigKey\r\n%% \r\n% Generate the signature with HMAC-SHA1 using Java as Twitty does.\r\n\r\nimport javax.crypto.Mac                                     % import Java libraries                  \r\nimport javax.crypto.spec.SecretKeySpec;                     % key spec methods\r\nimport org.apache.commons.codec.binary.Base64               % base64 codec\r\n\r\nalgorithm = strrep(sigMethod,'-','');                       % use HMAC-SHA1\r\nkey = SecretKeySpec(int8(sigKey), algorithm);               % get key\r\nmac = Mac.getInstance(algorithm);                           % message auth code\r\nmac.init(key);                                              % initialize mac\r\nmac.update(int8(sigStr));                                   % add base string\r\nparams.oauth_signature = ...                                % get signature \r\n    char(Base64.encodeBase64(mac.doFinal)'); \r\nparams = orderfields(params);                               % order fields\r\n\r\n%% Build the OAuth Header\r\n% With the signature added to the parameter, we can now build the OAuth header.\r\n\r\nkeys = fieldnames(params);                                  % get field names\r\nisOauth = contains(keys,'oauth');                           % oauth params\r\nkeys = cellfun(pctencode,keys,'UniformOutput',false);       % percent-encode keys\r\nkeys = string(keys) + '=\"';                                 % convert to string and add =\"\r\nvalues = struct2cell(params);                               % get field values\r\nvalues = cellfun(pctencode,values,'UniformOutput',false);   % percent-encode values\r\nvalues = string(values) +'\", ';                             % convert to string and add \",\r\noauth_header = join([keys, values],'');                     % join columns\r\noauth_header = oauth_header(isOauth,:);                     % keep OAuth params only\r\noauth_header = 'OAuth ' + join(oauth_header(:),'');         % convert to a single string\r\noauth_header{1}(end-1:end) = [];                            % remove extra \"\r\n\r\n%% Send HTTP Request\r\n% This is where everything we have done so far comes together and you get\r\n% to see how you can use HTTP Interface functions to send an HTTP request,\r\n% such as\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.uri-class.html\r\n% matlab.net.URI>,\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.http.requestmethod-class.html\r\n% matlab.net.http.RequestMethod>,\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.net.http.field.authenticatefield-class.html\r\n% matlab.net.http.field.AuthenticateField>, etc. I will just load the\r\n% previously saved response object here for reproducibility.\r\n\r\nimport matlab.net.URI;                                      % shorten lib calls\r\nimport matlab.net.http.RequestMethod;\r\nimport matlab.net.http.RequestMessage;\r\nimport matlab.net.http.field.AuthorizationField; \r\nimport matlab.net.http.field.ContentTypeField;\r\n\r\nconnURI = URI(base_url,queryStr);                           % create URI obj\r\nmethod = RequestMethod.(char(httpMethod));                  % create method obj\r\nauth = AuthorizationField('Authorization',oauth_header);    % create auth field obj\r\nct_value = 'application\/x-www-form-urlencoded';             % content type\r\ncontent_type = ContentTypeField(ct_value);                  % create content type obj\r\nheader = [auth content_type];                               % combine header fields objs\r\nrequest = RequestMessage(method,header);                    % create request obj\r\nresponse = send(request,connURI);                           % send http request\r\nload response                                               % load saved resp\r\ndisp(response)\r\n\r\n%% Process Response\r\n% The Twitter API returns its response in JSON format, but the HTTP\r\n% interface in MATLAB automatically parses it into a structure array. Now\r\n% you can process it with any familiar MATLAB functions of your choice. \r\n\r\nif string(fieldnames(response.Body.Data)) ~= 'errors'       % if not errors\r\n    s = response.Body.Data.statuses;                        % get tweets\r\n    if ~isempty(s)                                          % if not empty\r\n        tweets = cellfun(@(x) x.text, s, 'UniformOutput',0);% extract text\r\n        tweets = char(replace(tweets,char(10),' '));        % remove new line\r\n        tweets = string(tweets(:,1:70)) + '...';            % truncate tweets\r\n        disp(tweets)\r\n    end\r\nend\r\n\r\n%% Summary\r\n% In this post you saw how you can use HTTP Interface to access Twitter and\r\n% you now have a foundation to build your own tool to analyze Twitter feed.\r\n% HTTP interface feature offers other possibilities as well. For example, I\r\n% used it to get expanded urls from shortened urls. How would you use this\r\n% in your projects? Please share your ideas\r\n% <https:\/\/blogs.mathworks.com\/developer\/?p=871#respond here>.\r\n##### SOURCE END ##### 615df350a3df4f08a3fc066c27863844\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/developer\/files\/2017NASARocket.jpg\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" decoding=\"async\" loading=\"lazy\" \/><\/div><!--introduction--><p><i>Folks, today we have a nice collaboration between blogs. If you haven't already seen it, <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/951521\">Toshi Takeuchi<\/a> just posted on Loren's blog today to help you <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=2209\">analyze fake news<\/a>. However, he is also here today to teach you how to fish, so to speak, so you are empowered to use the protocol to transfer hypertext all over the interweb! Here's Toshi...<\/i>... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/developer\/2017\/02\/07\/connect-to-twitter-with-oauth-over-http-interface\/\">read more >><\/a><\/p>","protected":false},"author":90,"featured_media":878,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/871"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/comments?post=871"}],"version-history":[{"count":5,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/871\/revisions"}],"predecessor-version":[{"id":881,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/871\/revisions\/881"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media\/878"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media?parent=871"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/categories?post=871"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/tags?post=871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}