{"id":2103,"date":"2019-05-03T03:35:23","date_gmt":"2019-05-03T07:35:23","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=2103"},"modified":"2022-12-14T11:38:56","modified_gmt":"2022-12-14T16:38:56","slug":"matlab-and-blob-storage","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2019\/05\/03\/matlab-and-blob-storage\/","title":{"rendered":"MATLAB and Blob Storage"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>As a continuation from my <a href=\"https:\/\/blogs.mathworks.com\/developer\/2019\/03\/26\/the-bucket-list\/\">previous post<\/a>,  this post discusses the use of the MATLAB interface for Azure&#8482; Storage Blob.<\/p><p>The Windows Azure Blob service is scalable, cost-effective cloud storage for all your unstructured data.<\/p><p>MATLAB developers can already use these services with our shipping products as <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/import_export\/work-with-remote-data.html\">documented<\/a> by leveraging the <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/datastore.html\">datastore<\/a> function that allows easy read\/write access of data stored in Blob Storage (among other forms of remote data).<\/p><p>And in the same vein as the previous post, I will focus on additional interfaces that are available for developers. These interfaces are targeted at MATLAB developers who desire to exercise finer control over the storage service. Usually this need for control is driven by requirements to configure security settings, access control, and other features such as manipulating multiple blob types, table services, etc. typical in most enterprise applications.<\/p><p>MathWorks has released the <a href=\"https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob\">MATLAB Interface for Azure Storage Blob<\/a> on github to allow MATLAB developers to leverage the storage service on Azure.<\/p><p>To get started, clone this repository:<\/p><p>\r\n<pre>\r\n$ git clone --recursive https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob.git\r\n<\/pre>\r\n<\/p><p>The repository above contains MATLAB code that leverages the <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-java\">Azure SDK for Java<\/a>. This package uses certain third-party content which is licensed under separate license agreements. See the pom.xml file for third-party software downloaded at build time.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#36b6ec8d-c9ac-4da1-8544-d51c07391028\">Container operations<\/a><\/li><li><a href=\"#bacf3656-8197-4539-9060-0da2797a13f2\">Uploading a blob \/ data into a container<\/a><\/li><li><a href=\"#3afe61fe-c2a1-4191-b751-f7ac95b6862c\">Downloading a blob \/ data into MATLAB<\/a><\/li><\/ul><\/div><b>Build the underlying Java artifacts<\/b><p>You can build the underlying Java SDK using Maven and the process is straightforward:<\/p><p>\r\n<pre>\r\n$ cd matlab-azure-blob\/Software\/Java\/\r\n$ mvn clean package\r\n<\/pre>\r\n<\/p><p>On a successful build, a JAR archive is packaged and is made available to MATLAB by running the startup.m file<\/p><pre class=\"codeinput\">cd <span class=\"string\">matlab-azure-blob\/Software\/MATLAB<\/span>\r\nstartup\r\n<\/pre><p>All access to the blob services originates with a cloud storage account which allows the provisioning of one or more containers. These containers allow users to group a set of blobs. A blob can be either block blob, page blob or append blob types and can be used for storage of artifacts such as text, binary, or media.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/developer\/files\/Blob1.png\" alt=\"\"> <\/p><p>A good introduction to the service can be found at: <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/storage-introduction\" target=\"_blank\" rel=\"noopener\">https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/storage-introduction<\/a><\/p><p>The MATLAB interface to Azure Storage blobs To create a MATLAB client to work with the service, developers can provision an access key in the Azure portal and use it with the MATLAB interface to create a handle to the cloud storage account.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/developer\/files\/AzureStorageKeys.png\" alt=\"\"> <\/p><p>The MATLAB interface uses the same familiar interface as the underlying SDK so most of the solutions, documentation and community knowledge on online forums like StackOverflow should translate directly and easily.<\/p><p>In this case, the code to create a client looks like:<\/p><pre class=\"codeinput\"><span class=\"comment\">% Create a handle to the storage account<\/span>\r\naz = azure.storage.CloudStorageAccount;\r\naz.AccountName = <span class=\"string\">'myaccountname'<\/span>;\r\naz.AccountKey  = <span class=\"string\">'ABCDEFGH*****[REDACTED]****ABCDEFGH'<\/span>;\r\naz.connect();\r\n\r\n<span class=\"comment\">% Create a handle to the CloudBlobClient<\/span>\r\nazClient = azure.storage.blob.CloudBlobClient(az);\r\n<\/pre><h4>Container operations<a name=\"36b6ec8d-c9ac-4da1-8544-d51c07391028\"><\/a><\/h4><p>The interfaces to create, list and configure containers is equally straightforward.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Create a container and list all existing containers<\/span>\r\nazContainer = azure.storage.blob.CloudBlobContainer(azClient,<span class=\"string\">'testcontainer'<\/span>);\r\nazContainer.createIfNotExists();\r\ncontainers = azClient.listContainers();\r\n\r\n<span class=\"comment\">% Configure a container for public access<\/span>\r\nperm = azure.storage.blob.BlobContainerPermissions;\r\nperm.AccessType = <span class=\"string\">'CONTAINER'<\/span>;  <span class=\"comment\">% Container-level public access<\/span>\r\nazContainer.uploadPermissions(perm);\r\n<\/pre><h4>Uploading a blob \/ data into a container<a name=\"bacf3656-8197-4539-9060-0da2797a13f2\"><\/a><\/h4><p>Blobs can be content of any nature. To generate some data in MATLAB and save it to newly created container:<\/p><pre class=\"codeinput\"><span class=\"comment\">% Create some random data<\/span>\r\nsampleData = rand(1000,1000);    <span class=\"comment\">% Approx 7MB<\/span>\r\nsave <span class=\"string\">SampleData<\/span> <span class=\"string\">sampleData<\/span>;\r\n\r\n<span class=\"comment\">% Uploading the data to a previously created container, create a blob handle (merely a reference) and upload.<\/span>\r\nblob = azContainer.getBlockBlobReference(which(<span class=\"string\">'SampleData.mat'<\/span>));\r\nblob.upload();\r\n<\/pre><p>And that is it, our first piece of data has been uploaded to the Cloud and stored on the Azure Blob Storage service.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/developer\/files\/SampleStorage.png\" alt=\"\"> <\/p><p>The interface is fully vectorized, so it is possible to upload an entire directory of files by invoking the upload method on a collection of configured blobs.<\/p><h4>Downloading a blob \/ data into MATLAB<a name=\"3afe61fe-c2a1-4191-b751-f7ac95b6862c\"><\/a><\/h4><p>Listing the contents of a container and downloading from the Blob storage service to MATLAB is equally straightforward.<\/p><pre class=\"codeinput\"><span class=\"comment\">% List all existing blobs<\/span>\r\nblobList = azContainer.listBlobs();\r\n\r\n<span class=\"comment\">% Download a particular file from cloud service into the current directory<\/span>\r\nblob = azure.storage.blob.CloudBlockBlob(azContainer, <span class=\"string\">'mydir1\/SampleData.mat'<\/span>);\r\nblobList.download(); <span class=\"comment\">% can accept an optional target directory argument<\/span>\r\n<\/pre><p>This basic example just scratches the surface of the possibilities of the interface. The package contains numerous features to work with common client workflows such as controlling access using Shared Access Signatures (SAS), etc.<\/p><p>And for clients that are using other types of blobs such as Table blobs, the interface provides support as described in the <a href=\"https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob\/blob\/master\/Documentation\/BasicUsageTable.md\">relevant sections of the documentation<\/a>.<\/p><p>These interfaces enable new and powerful ways to extend MATLAB to access cloud based storage systems to store, partition and analyze data. This extended functionality takes on special significance when used with our reference architectures of <a href=\"https:\/\/mathworks.com\/cloud\">running the MathWorks products on the public cloud systems<\/a> like Azure to improve data access and analysis performance.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/developer\/files\/BlockBlobBlog.png\" alt=\"\"> <\/p><p>That concludes my brief blurb about block blob on the blog... try saying that aloud fast. Or as the old meme goes - On the internet, nobody knows you're a blob (with apologies to <a href=\"https:\/\/en.wikipedia.org\/wiki\/On_the_Internet,_nobody_knows_you%27re_a_dog\" target=\"_blank\" rel=\"noopener\">Peter Steiner<\/a>).<\/p><p>In closing, I wanted to share this with you, the reader, to underline the fact that MATLAB developers can leverage these capabilities to build truly impressive data analytics systems that are cloud capable and run at scale. All of this becomes possible within the comfort of their time-tested MATLAB environments and workflows.<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_01616a60012240eda86517ee1ef0a9a4() {\r\n        \/\/ Remember the title so we can use it on the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='01616a60012240eda86517ee1ef0a9a4 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 01616a60012240eda86517ee1ef0a9a4';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2019 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_01616a60012240eda86517ee1ef0a9a4()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2019a<br><\/p><\/div><!--\r\n01616a60012240eda86517ee1ef0a9a4 ##### SOURCE BEGIN #####\r\n%%\r\n% *MATLAB and Blob Storage*\r\n% \r\n% In this blog post, I discuss the use of the MATLAB interface for Azure\u00e2\u201e\u00a2\r\n% Storage Blob. This post is loosely a continuation from my \r\n% <https:\/\/blogs.mathworks.com\/developer\/2019\/03\/26\/the-bucket-list\/ previous post>\r\n% which detailed use of MATLAB interface to the similar object storage system from AWS.\r\n% \r\n% The Windows Azure Blob service is scalable, cost-effective cloud storage \r\n% for all your unstructured data. \r\n% \r\n% MATLAB developers can already use these services with our shipping R2018b\/19a \r\n% products as <https:\/\/www.mathworks.com\/help\/matlab\/import_export\/work-with-remote-data.html documented> \r\n% by leveraging the <https:\/\/www.mathworks.com\/help\/matlab\/ref\/datastore.html datastore>\r\n% function that allows easy read\/write access of data stored in Blob Storage \r\n% (among other forms of remote data).\r\n% \r\n% And in the same vein as the previous post, the additional interfaces\r\n% are available for developers. These interfaces are targeted at MATLAB developers who \r\n% desire to exercise finer control of the service. Usually these are driven by \r\n% requirements to configure security settings, access control,\r\n% and other features such as manipulating multiple blob types, table services, etc.\r\n% typical in most enterprise applications.\r\n% \r\n% MathWorks has released the \r\n% <https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob MATLAB Interface for Azure Storage Blob> on\r\n% github to allow MATLAB developers to leverage this storage service on\r\n% Azure. \r\n% \r\n% To get started, clone this repository:\r\n% \r\n% <html>\r\n% <pre>\r\n% $ git clone REPLACE_WITH_DASH_DASHrecursive https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob.git\r\n% <\/pre>\r\n% <\/html>\r\n% \r\n% The repository above contains MATLAB code that leverages the <https:\/\/github.com\/Azure\/azure-sdk-for-java Azure SDK for Java>. \r\n% This package uses certain third-party content which is licensed under separate license agreements. \r\n% See the pom.xml file for third-party software downloaded at build time.\r\n%\r\n%%\r\n% *Build the underlying Java artifacts*\r\n% You can build the underlying Java SDK using Maven and the process is straightforward:\r\n%\r\n% <html>\r\n% <pre>\r\n% $ cd matlab-azure-blob\/Software\/Java\/\r\n% $ mvn clean package\r\n% <\/pre>\r\n% <\/html>\r\n%\r\n% On a successful build, a JAR archive is packaged and is made available to MATLAB by running the startup.m file\r\n% \r\ncd matlab-aws-s3\/Software\/MATLAB\r\nstartup\r\n\r\n%%\r\n% All access to the blob services originates with a cloud storage account which\r\n% allows the provisioning of one or more containers. These containers allow users to group\r\n% a set of blobs. A blob can be either block blob, page blob or append blob type\r\n% and can be used for storage of artifacts such as text, binary, or media.\r\n% \r\n% <<Blob1.png>>\r\n% \r\n% A good introduction to the service can be found at:\r\n% https:\/\/docs.microsoft.com\/en-us\/azure\/storage\/common\/storage-introduction\r\n%\r\n%% \r\n% The MATLAB interface to Azure Storage blobs\r\n% To create a MATLAB client to work with the service, developers can provision\r\n% an access key in the Azure portal and use it with the MATLAB interface to\r\n% create a handle to the cloud storage account. \r\n% \r\n% <<AzureStorageKeys.png>>\r\n% \r\n% The MATLAB interface uses the same familiar interface as the underlying\r\n% SDK so most of the solutions, documentation and community knowledge on online forums \r\n% like StackOverflow should translate directly and easily.\r\n% \r\n% In this case, the code to create a client looks like:\r\n\r\n% Create a handle to the storage account\r\naz = azure.storage.CloudStorageAccount;\r\naz.AccountName = 'myaccountname';\r\naz.AccountKey  = 'ABCDEFGH*****[REDACTED]****ABCDEFGH';\r\naz.connect();\r\n\r\n% Create a handle to the CloudBlobClient\r\nazClient = azure.storage.blob.CloudBlobClient(az);\r\n\r\n%% Container operations\r\n% The interfaces to create, list and configure containers is equally\r\n% straightforward.\r\n\r\n% Create a container and list all existing containers\r\nazContainer = azure.storage.blob.CloudBlobContainer(azClient,'testcontainer');\r\nazContainer.createIfNotExists();\r\ncontainers = azClient.listContainers();\r\n\r\n% Configure a container for public access\r\nperm = azure.storage.blob.BlobContainerPermissions;\r\nperm.AccessType = 'CONTAINER';  % Container-level public access\r\nazContainer.uploadPermissions(perm);\r\n\r\n%% Uploading a blob\/data into a container\r\n% Blobs can be content of any nature. To generate some data in MATLAB and \r\n% save it to the newly created container as follows:\r\n\r\n% Create some random data\r\nsampleData = rand(1000,1000);    % Approx 7MB\r\nsave SampleData sampleData;\r\n\r\n% Uploading the data to a previously created container, create a blob handle (merely a reference) and upload.\r\nblob = azContainer.getBlockBlobReference(which('SampleData.mat'));\r\nblob.upload();\r\n\r\n%%\r\n% And that is it, our first piece of data has been uploaded to the Cloud\r\n% and stored on the Azure Blob Storage service. \r\n% \r\n% <<SampleStorage.png>>\r\n% \r\n% The interface is fully vectorized, so it is possible to run the upload on an entire collection\r\n% of files by invoking the upload method on a collection of configured blobs.\r\n%\r\n%% Downloading a blob\/data into MATLAB\r\n% Listing the contents of a container and downloading from the Blob storage service to MATLAB is equally\r\n% straightforward.\r\n\r\n% List all existing blobs\r\nblobList = azContainer.listBlobs();\r\n\r\n% Download a particular file from cloud service into the current directory\r\nblob = azure.storage.blob.CloudBlockBlob(azContainer, 'mydir1\/SampleData.mat');\r\nblobList.download(); % can accept an optional target directory argument\r\n\r\n%%\r\n% This basic example just scratches the surface of the possibilities of the\r\n% interface. The package contains numerous features to work with common\r\n% client workflows such as controlling access using Shared Access Signatures (SAS), etc.\r\n% \r\n% And for clients that are using other types of blobs such as Table blobs, the support is\r\n% can be found in the <https:\/\/github.com\/mathworks-ref-arch\/matlab-azure-blob\/blob\/master\r\n% \/Documentation\/BasicUsageTable.md relevant sections of the documentation>.\r\n%\r\n% These interfaces enable new and powerful ways to extend\r\n% MATLAB to access cloud-based storage systems to store, partition and\r\n% analyze data. This extended functionality takes on special significance when used\r\n% with our reference architectures of <https:\/\/mathworks.com\/cloud running the MathWorks products on the\r\n% public cloud systems> like Azure to improve data access and analysis performance.\r\n% \r\n% <<BlockBlobBlog.png>>\r\n% \r\n% In closing, I wanted to share this with you, the reader, to underline the \r\n% fact that MATLAB developers can leverage these capabilities to building truly impressive data\r\n% analytics systems that are cloud capable and run at scale. All of this\r\n% becomes possible within the comfort of their time-tested MATLAB\r\n% environments and workflows.\r\n\r\n##### SOURCE END ##### 01616a60012240eda86517ee1ef0a9a4\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/developer\/files\/Blob1.png\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"StorageAccountsContainersAndBlobs\" decoding=\"async\" loading=\"lazy\" \/><\/div><!--introduction--><p>As a continuation from my <a href=\"https:\/\/blogs.mathworks.com\/developer\/2019\/03\/26\/the-bucket-list\/\">previous post<\/a>,  this post discusses the use of the MATLAB interface for Azure&#8482; Storage Blob.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/developer\/2019\/05\/03\/matlab-and-blob-storage\/\">read more >><\/a><\/p>","protected":false},"author":135,"featured_media":2109,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22,10],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/2103"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/users\/135"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/comments?post=2103"}],"version-history":[{"count":22,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/2103\/revisions"}],"predecessor-version":[{"id":2960,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/2103\/revisions\/2960"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media\/2109"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media?parent=2103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/categories?post=2103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/tags?post=2103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}