{"id":722,"date":"2022-12-20T06:41:34","date_gmt":"2022-12-20T11:41:34","guid":{"rendered":"https:\/\/blogs.mathworks.com\/matlab\/?p=722"},"modified":"2022-12-20T06:57:29","modified_gmt":"2022-12-20T11:57:29","slug":"add-two-lines-to-your-matlab-code-to-make-it-work-with-big-data","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/matlab\/2022\/12\/20\/add-two-lines-to-your-matlab-code-to-make-it-work-with-big-data\/","title":{"rendered":"Add two lines to your MATLAB code to make it work with big data"},"content":{"rendered":"
\r\n
Today's guest blogger is Harald Brunnhofer, a Principal Training Engineer at MathWorks. On top of getting thousands of users started with MATLAB or taking their MATLAB skills to the next level, he is the content developer of the <\/span>Accelerating and Parallelizing MATLAB Code<\/span><\/a> and <\/span>Processing Big Data with MATLAB<\/span><\/a> training courses. <\/span><\/div>\r\n
In an earlier post, Mike gave an overview of MATLAB's High Performance Computing (HPC) and Big Data datatypes<\/a>. Today, I'll be showing you how you can use tall arrays to move from small to big data with minimal changes to your code.<\/div>\r\n
\"Big data\" is becoming more and more relevant in a variety of industries and applications. For our purpose, this is data that is (far) too large to fit into the memory of a single computer: tens of gigabytes, possibly terabytes or even petabytes (= 1024 terabytes) of data. So what are you going to do if you are getting out-of-memory errors or know very well that you will get these before you even try?<\/div>\r\n
\r\n
Table of Contents<\/span><\/div>\r\n
Some general advice for large datasets\r\n<\/a>Datastores: Good for big and small data\r\n<\/a> Set up the datastore\r\n<\/a> Import and inspect the data\r\n<\/a> Perform the analysis\r\n<\/a>Tall Arrays: An easy lazy evaluation framework for handling big data\r\n<\/a> Generating some big data for experimenting\r\n<\/a> Set up the datastore\r\n<\/a> Setting up parallel resources\r\n<\/a> Import the data\r\n<\/a> Perform the analysis\r\n<\/a> Getting the results\r\n<\/a> A slightly different problem\r\n<\/a> Delete test big data (optional)\r\n<\/a>Summary and Resources\r\n<\/a>Supporting functions<\/a><\/div>\r\n<\/div>\r\n
<\/div>\r\n

Some general advice for large datasets<\/h2>\r\n
Even before data gets \"big\", it may be of interest to save memory, time, or both by<\/div>\r\n