{"id":9382,"date":"2018-01-19T09:13:04","date_gmt":"2018-01-19T14:13:04","guid":{"rendered":"https:\/\/blogs.mathworks.com\/pick\/?p=9382"},"modified":"2018-01-19T09:14:11","modified_gmt":"2018-01-19T14:14:11","slug":"ugly-data-app","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/pick\/2018\/01\/19\/ugly-data-app\/","title":{"rendered":"Ugly Data App"},"content":{"rendered":"<div xmlns:mwsh=\"https:\/\/www.mathworks.com\/namespace\/mcode\/v1\/syntaxhighlight.dtd\" class=\"content\">\n   <introduction><\/p>\n<p><i>Richard is a Consultant at MathWorks focused on the Embedded Coder product for code generation, primarily in the Aerospace<br \/>\n            industry.<\/i><\/p>\n<p><a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/1016860-richard-ruff\">Richard&#8217;s<\/a> pick this week is <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app\">Ugly Data App<\/a> by <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/4456782-adam-sifounakis\">Adam Sifounakis<\/a>.\n      <\/p>\n<p>   <\/introduction><\/p>\n<h3>Contents<\/h3>\n<div>\n<ul>\n<li><a href=\"#1\">Pick<\/a><\/li>\n<li><a href=\"#2\">Enhancements<\/a><\/li>\n<li><a href=\"#4\">Comments<\/a><\/li>\n<\/ul><\/div>\n<h3>Pick<a name=\"1\"><\/a><\/h3>\n<p>My pick this week is <b><a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app\">Ugly Data App<\/a> &#8211; a tool to clean up data<\/b> .\n   <\/p>\n<p>This submission is a great example of the types of complex User Interfaces that can be created using MATLAB&#8217;s App Designer.<br \/>\n       The tool itself allows the user to bring in data from the MATLAB workspace and then operate on it in a number of ways.\n   <\/p>\n<p>The first option is to handle missing values.  The user can specify how the missing data should be handled by selecting a<br \/>\n      fill method from the list such as Linear or Spline.  The tool also allows the user to constrain the data by specifying thresholds<br \/>\n      at which the data will be replaced with NaNs.  As an example, the data may only be valid for values greater than zero.  This<br \/>\n      allows you to eliminate those erroneous data points.\n   <\/p>\n<p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/pick\/Sean\/main_UglyDataApp\/MissingValues.png\"> <\/p>\n<p>The next option is to handle any outliers in the data.  The user can specify the outlier method to use from a list and the<br \/>\n      corresponding parameters associated with the method such as the threshold factor.  The plot is updated to show the boundaries<br \/>\n      of the outliers.  Similar to the Missing Data, the user can specify a fill method for the outliers from a list.\n   <\/p>\n<p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/pick\/Sean\/main_UglyDataApp\/Outliers.png\"> <\/p>\n<p>The third option allows the user to smooth the data.  The smoothing method can be selected from a list and the smoothing parameters<br \/>\n      set.\n   <\/p>\n<p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/pick\/Sean\/main_UglyDataApp\/Smoothing.png\"> <\/p>\n<p>Once the data has been &#8220;cleaned&#8221;, it can be exported to either MATLAB&#8217;s Workspace or to an Excel file via the &#8220;Export Data&#8221;<br \/>\n      menu item.  There are also menu items to specify &#8220;Plot Options&#8221;, provide &#8220;Feedback&#8221; to Adam, and to get &#8220;Help&#8221; for the App.\n   <\/p>\n<p>However, the feature of this App that sets it apart is the ability to &#8220;Generate Code&#8221;.  This menu item allows the user to<br \/>\n      automatically create a MATLAB script or function that performs the same steps.  This allows the user to &#8220;set-up&#8221; the data<br \/>\n      cleaning once, and then easily apply it to as many data sets as desired.  The resulting MATLAB code follows the same steps<br \/>\n      that the user did manually.  As such, the user needs to keep this in mind when working with the data.  You probably don&#8217;t<br \/>\n      want to insert NaNs after having filled the NaNs. The order of the steps can be seen in the lower left corner in the &#8220;Code<br \/>\n      Generation&#8221; box.\n   <\/p>\n<h3>Enhancements<a name=\"2\"><\/a><\/h3>\n<p>There are some enhancements that I would like to see Adam make to this App:<\/p>\n<div>\n<ul>\n<li>Provide &#8220;zoom&#8221; capability on the plot to allow the user to focus in on a section of the data.<\/li>\n<li>Provide more documentation on the usage of the App.  A set of examples highlighting the capabilities would be beneficial.<\/li>\n<li>Provide an option to show a subplot of the difference between the original data and the adjusted data.<\/li>\n<li>Insert a marker for any NaNs found in the raw data and not just those that are byproducts of setting boundaries.<\/li>\n<\/ul><\/div>\n<h3>Comments<a name=\"4\"><\/a><\/h3>\n<p>As stated before, this is a great example of the custom User Interfaces you can build in MATLAB.  If you have test data that<br \/>\n      contains dropouts or outliers, this App allows you to manually adjust the data.  However, the real benefit is the ability<br \/>\n      to then create a custom MATLAB script\/function to perform the same adjustments &#8211; which allows it to be applied to many sets<br \/>\n      of data. Give it a try and let us know what you think <a href=\"https:\/\/blogs.mathworks.com\/pick\/?p=9382#respond\">here<\/a> or leave a <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app#comments\">message<\/a> for Adam.\n   <\/p>\n<p><script language=\"JavaScript\">\n<!--\n\n    function grabCode_aa2bea7431a6400eb27ba7a68f8c7720() {\n        \/\/ Remember the title so we can use it in the new page\n        title = document.title;\n\n        \/\/ Break up these strings so that their presence\n        \/\/ in the Javascript doesn't mess up the search for\n        \/\/ the MATLAB code.\n        t1='aa2bea7431a6400eb27ba7a68f8c7720 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\n        t2='##### ' + 'SOURCE END' + ' #####' + ' aa2bea7431a6400eb27ba7a68f8c7720';\n    \n        b=document.getElementsByTagName('body')[0];\n        i1=b.innerHTML.indexOf(t1)+t1.length;\n        i2=b.innerHTML.indexOf(t2);\n \n        code_string = b.innerHTML.substring(i1, i2);\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\n\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \n        \/\/ in the XML parser.\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\n        \/\/ doesn't go ahead and substitute the less-than character. \n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\n\n        author = 'Richard Ruff';\n        copyright = 'Copyright 2018 The MathWorks, Inc.';\n\n        w = window.open();\n        d = w.document;\n        d.write('\n\n<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add author and copyright lines at the bottom if specified.\r\n        if ((author.length > 0) || (copyright.length > 0)) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (author.length > 0) {\r\n                d.writeln('% _' + author + '_');\r\n            }\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\n\n\\n');\n      \n      d.title = title + ' (MATLAB code)';\n      d.close();\n      }   \n      \n-->\n<\/script><\/p>\n<p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><a href=\"javascript:grabCode_aa2bea7431a6400eb27ba7a68f8c7720()\"><span style=\"font-size: x-small;        font-style: italic;\">Get<br \/>\n            the MATLAB code<br \/>\n            <noscript>(requires JavaScript)<\/noscript><\/span><\/a><\/p>\n<p>      Published with MATLAB&reg; R2017b<\/p>\n<\/div>\n<p><!--\naa2bea7431a6400eb27ba7a68f8c7720 ##### SOURCE BEGIN #####\n%% Ugly Data App\n%\n% Posted by *Richard Ruff* , January 19, 2018\n%\n% _Richard is a Consultant at MathWorks focused on the Embedded Coder\n% product for code generation, primarily in the Aerospace industry._\n% \n% <https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/1016860-richard-ruff\n% Richard's> pick this week is\n% <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app\n% Ugly Data App> by\n% <https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/4456782-adam-sifounakis\n% Adam Sifounakis>.\n% \n%% Pick\n% My pick this week is\n% *<https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app\n% Ugly Data App> - a tool to clean up data* .\n%\n% This submission is a great example of the types of complex User\n% Interfaces that can be created using MATLAB's App Designer.  The tool\n% itself allows the user to bring in data from the MATLAB workspace and\n% then operate on it in a number of ways.\n%\n% The first option is to handle missing values.  The user can specify how\n% the missing data should be handled by selecting a fill method from the\n% list such as Linear or Spline.  The tool also allows the user to\n% constrain the data by specifying thresholds at which the data will be\n% replaced with NaNs.  As an example, the data may only be valid for values\n% greater than zero.  This allows you to eliminate those erroneous data\n% points.\n%\n% <<C:\\MATLAB\\PCTransferMATLAB\\PickOfWeek\\UglyDataApp\\Missingvalues.png>>\n%\n% The next option is to handle any outliers in the data.  The user can\n% specify the outlier method to use from a list and the corresponding\n% parameters associated with the method such as the threshold factor.  The\n% plot is updated to show the boundaries of the outliers.  Similar to the\n% Missing Data, the user can specify a fill method for the outliers from a\n% list.\n%\n% <<C:\\MATLAB\\PCTransferMATLAB\\PickOfWeek\\UglyDataApp\\Outliers.png>>\n%\n% The third option allows the user to smooth the data.  The smoothing\n% method can be selected from a list and the smoothing parameters set.\n%\n% <<C:\\MATLAB\\PCTransferMATLAB\\PickOfWeek\\UglyDataApp\\Smoothing.png>>\n%\n% Once the data has been \"cleaned\", it can be exported to either MATLAB's\n% Workspace or to an Excel file via the \"Export Data\" menu item.  There are\n% also menu items to specify \"Plot Options\", provide \"Feedback\" to Adam,\n% and to get \"Help\" for the App.\n%\n% However, the feature of this App that sets it apart is the ability to\n% \"Generate Code\".  This menu item allows the user to automatically create\n% a MATLAB script or function that performs the same steps.  This allows\n% the user to \"set-up\" the data cleaning once, and then easily apply it to\n% as many data sets as desired.  The resulting MATLAB code follows the same\n% steps that the user did manually.  As such, the user needs to keep this\n% in mind when working with the data.  You probably don't want to insert\n% NaNs after having filled the NaNs. The order of the steps can be seen in\n% the lower left corner in the \"Code Generation\" box.\n%\n\n%% Enhancements\n%\n% There are some enhancements that I would like to see Adam make to this\n% App:\n%%\n% \n% * Provide \"zoom\" capability on the plot to allow the user to focus in on\n% a section of the data.\n% * Provide more documentation on the usage of the App.  A set of examples\n% highlighting the capabilities would be beneficial.\n% * Provide an option to show a subplot of the difference between the\n% original data and the adjusted data.\n% * Insert a marker for any NaNs found in the raw data and not just those\n% that are byproducts of setting boundaries.\n% \n%% Comments\n% \n% As stated before, this is a great example of the custom User Interfaces\n% you can build in MATLAB.  If you have test data that contains dropouts or\n% outliers, this App allows you to manually adjust the data.  However, the\n% real benefit is the ability to then create a custom MATLAB\n% script\/function to perform the same adjustments - which allows it to be\n% applied to many sets of data. Give it a try and let us know what you\n% think <https:\/\/blogs.mathworks.com\/pick\/?p=9382#respond here> or leave a\n% <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/65343-ugly-data-app#comments\n% message> for Adam.\n%\n \n\n##### SOURCE END ##### aa2bea7431a6400eb27ba7a68f8c7720\n--><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/images\/pick\/Sean\/main_UglyDataApp\/MissingValues.png\" onError=\"this.style.display ='none';\" \/><\/div>\n<p>Richard is a Consultant at MathWorks focused on the Embedded Coder product for code generation, primarily in the Aerospace<br \/>\n            industry.<br \/>\nRichard&#8217;s pick this week is Ugly Data App&#8230; <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/pick\/2018\/01\/19\/ugly-data-app\/\">read more >><\/a><\/p>\n","protected":false},"author":131,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/9382"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/users\/131"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/comments?post=9382"}],"version-history":[{"count":4,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/9382\/revisions"}],"predecessor-version":[{"id":9390,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/9382\/revisions\/9390"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/media?parent=9382"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/categories?post=9382"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/tags?post=9382"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}