{"id":1349,"date":"2016-02-15T08:54:13","date_gmt":"2016-02-15T13:54:13","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=1349"},"modified":"2018-11-20T15:44:47","modified_gmt":"2018-11-20T20:44:47","slug":"reproducibility-musings-hey-do-that-again","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2016\/02\/15\/reproducibility-musings-hey-do-that-again\/","title":{"rendered":"Reproducibility Musings &#8211; Hey, do that again!"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>Today I have a guest post from <a href=\"mailto:lisak@mathworks.com\">Lisa Kempler<\/a>, MATLAB Community Strategist, MathWorks. Lisa works with MATLAB communities in domains such as geophysics, oceanography, audiology, and more, helping users developing MATLAB based tools, and creating resources for teaching and research with MATLAB. A primary goal is to have MATLAB users share their tools and best practices more widely within their communities, enhancing their use of MATLAB, and, in turn, accelerating their work. This blog talks about some efforts in research that support that sharing objective.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#96f8d285-9c79-41b4-9187-20f43aefc00e\">What is Reproducibility?<\/a><\/li><li><a href=\"#af1ce0cf-3058-4260-b6c6-332f3a8d6eae\">Why is it important?<\/a><\/li><li><a href=\"#05943774-fade-4ca3-9330-1dee4ac054f2\">Getting Consensus on Reproducible Research Needs and Approaches<\/a><\/li><li><a href=\"#65dd954b-5251-4fbd-b55e-82911f47293b\">Individual Researcher Efforts and MATLAB Related Tools and Capabilities<\/a><\/li><li><a href=\"#9ea46e79-b23b-46b3-9897-a1cf2acdc0f9\">Your Turn<\/a><\/li><li><a href=\"#1f1e2039-8081-455c-aa88-d075872f82d2\">Kevin Moerman's Affiliations<\/a><\/li><\/ul><\/div><h4>What is Reproducibility?<a name=\"96f8d285-9c79-41b4-9187-20f43aefc00e\"><\/a><\/h4><p>There&#8217;s been lots of buzz around the topic of Reproducible Research. Proponents as well as the researchers who would be impacted by new reproducibility expectations are raising a lot of questions, questions about<\/p><div><ul><li>Definition and Scope:<\/li><\/ul><\/div><pre>  - Simply put, what does Reproducible Research mean?\r\n  - What does it encompass &#8211; Data? Software? Hardware?\r\n    Experimental and field physical environment setup?<\/pre><div><ul><li>Guidelines and Oversight:<\/li><\/ul><\/div><pre>  - Who will develop the standards?\r\n  - How will they be enforced?<\/pre><div><ul><li>Usefulness and Practicality:<\/li><\/ul><\/div><pre>  - How will scientists fund the additional time and resources to adhere\r\n    to the new requirements?\r\n  - Will anyone actually try to reproduce the research using all this\r\n    new machinery? How often? Enough to make it worth the effort?<\/pre><p>At its most basic level, the idea of reproducible research is that someone in the future, wanting to recreate the results of your work, could. The theory is that Reproducibility will enable more complete understanding and use of published research, that enabling future researchers to reuse and reproduce the work of those who preceded them will be more efficient, avoiding reinventing wheels, and save us all collectively time to solutions. Such a practical objective seems desirable at face value. However, this first-order impact is not the only driver for organizations pushing for Reproducibility.<\/p><h4>Why is it important?<a name=\"af1ce0cf-3058-4260-b6c6-332f3a8d6eae\"><\/a><\/h4><p>An overarching objective of the Reproducibility movement is a desire for reliable research. In other words, research sponsors and consumers want to know that the reported research results are accurate and have been validated. By creating and enforcing new research and publishing best practices around Reproducibility, sponsoring organizations, such as IEEE, NIH, and NSF, are looking to significantly reduce future retractions, including ones resulting from plagiarism. The hope is that if those submitting papers for publication are required to make their research inputs, methodologies, and outputs more transparent, the research findings, as a result, will necessarily be more reliable. More reliable conclusions, in turn, the theory goes, will result in fewer retractions. As an added bonus, this reproducible research can then be reused by other researchers.<\/p><p>This may seem like a lot of work just to avoid retractions. A shocking fact is that there are hundreds to even thousands of retractions per year. One 2014 <a href=\"http:\/\/phys.org\/news\/2014-02-science-publisher-gibberish-papers.html\">article in phys.org<\/a> reported finding 16 articles in their archives that were actually computer-generated; yes, they were produced by a machine, not a human. Following that, IEEE found 100 more in their archives.<\/p><h4>Getting Consensus on Reproducible Research Needs and Approaches<a name=\"05943774-fade-4ca3-9330-1dee4ac054f2\"><\/a><\/h4><p>Various organizations have held meetings on Reproducibility with the goal of reaching some consensus on best practices and next steps. Representatives from MathWorks have participated in multiple of these. As examples, Brown University&#8217;s ICERM held a workshop &#8211; <a href=\"https:\/\/icerm.brown.edu\/tw12-5-rcem\">Reproducibility in Computational and Experimental Mathematics<\/a> &#8211; back in 2012, and Loren attended. And this past December, there was a <a href=\"https:\/\/agu.confex.com\/agu\/fm15\/meetingapp.cgi\/Session\/8417\">session dedicated to Sustainable Software<\/a> at the American Geophysical Union (AGU) Fall Meeting where I delivered a talk. Multiple other discussions have been held on related topics with still more to come; it&#8217;s a bold endeavor with many players, not to mention that accessible and usable cyber-infrastructure will need to be part of any solution that gets buy-in and use by the target user population.<\/p><h4>Individual Researcher Efforts and MATLAB Related Tools and Capabilities<a name=\"65dd954b-5251-4fbd-b55e-82911f47293b\"><\/a><\/h4><p>Meanwhile, some researchers are forging ahead on their own, taking it upon themselves to make their research reproducible and, at the same time, more transparent for readers and other consumers. One such researcher is Kevin Moerman of Trinity College and the MIT Biomechatronics Media Lab. In a September 2015 blog post, he describes the challenges in academia that led him to combine MATLAB and open science. <a href=\"https:\/\/thewinnower.com\/papers\/2436-open-source-publishing-to-boost-your-career\">In his post, he essentially documents a recipe<\/a> that other researchers can follow to share code and make their work more transparent.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2016\/gibbonsSmall.jpg\" alt=\"\"> <\/p><p>A MATLAB feature that enabled Kevin to make the toolbox even more accessible is <a href=\"https:\/\/blogs.mathworks.com\/community\/2014\/04\/18\/file-exchange-now-hosts-github\/\">GitHub peering<\/a>. That means that Kevin contributed the toolbox to <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/\">MATLAB File Exchange<\/a> and set the source of the download to GitHub, allowing him to offer two entry-points to the toolbox, with only one set of source code.<\/p><p>In his post, Kevin concludes that developing the <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/48208-gibboncode-gibbon?s_tid=srchtitle\">Gibbon Toolbox<\/a> and making it freely available has been a huge boost for his career. According to Kevin, &#8220;Publishing GIBBON open source has maximized the impact of my work on an international level&#8221;.<\/p><h4>Your Turn<a name=\"9ea46e79-b23b-46b3-9897-a1cf2acdc0f9\"><\/a><\/h4><p>How do you make your research results available to your community? Let us know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=1349#respond\">here<\/a>.<\/p><h4>Kevin Moerman's Affiliations<a name=\"1f1e2039-8081-455c-aa88-d075872f82d2\"><\/a><\/h4><p>1.  Trinity Centre for Bioengineering, University of Dublin, Trinity College, Dublin, Ireland<\/p><p>2. Biomechatronics, Media Lab., Massachusetts Institute of Technology, Cambridge MA, USA<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_9597f0dbda21483f99c3be9b60097cb5() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='9597f0dbda21483f99c3be9b60097cb5 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 9597f0dbda21483f99c3be9b60097cb5';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2016 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_9597f0dbda21483f99c3be9b60097cb5()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2015b<br><\/p><\/div><!--\r\n9597f0dbda21483f99c3be9b60097cb5 ##### SOURCE BEGIN #####\r\n%% Reproducibility Musings - Hey, do that again!\r\n% Today I have a guest post from <mailto:lisak@mathworks.com Lisa Kempler>, MATLAB\r\n% Community Strategist, MathWorks. Lisa works with MATLAB communities in\r\n% domains such as geophysics, oceanography, audiology, and more, helping\r\n% users developing MATLAB based tools, and creating resources for teaching\r\n% and research with MATLAB. A primary goal is to have MATLAB users share\r\n% their tools and best practices more widely within their communities,\r\n% enhancing their use of MATLAB, and, in turn, accelerating their work.\r\n% This blog talks about some efforts in research that support that sharing objective.\r\n%% What is Reproducibility?\r\n% There\u00e2\u20ac&#x2122;s been lots of buzz around the topic of Reproducible Research.\r\n% Proponents as well as the researchers who would be impacted by new\r\n% reproducibility expectations are raising a lot of questions, questions\r\n% about \r\n%\r\n% * Definition and Scope: \r\n%\r\n%    - Simply put, what does Reproducible Research mean?\r\n%    - What does it encompass \u00e2\u20ac\u201c Data? Software? Hardware? \r\n%      Experimental and field physical environment setup?\r\n%\r\n% * Guidelines and Oversight: \r\n%\r\n%    - Who will develop the standards? \r\n%    - How will they be enforced? \r\n%\r\n% * Usefulness and Practicality:\r\n%\r\n%    - How will scientists fund the additional time and resources to adhere \r\n%      to the new requirements? \r\n%    - Will anyone actually try to reproduce the research using all this \r\n%      new machinery? How often? Enough to make it worth the effort?\r\n% \r\n% At its most basic level, the idea of reproducible research is that\r\n% someone in the future, wanting to recreate the results of your work,\r\n% could. The theory is that Reproducibility will enable more complete\r\n% understanding and use of published research, that enabling future\r\n% researchers to reuse and reproduce the work of those who preceded them\r\n% will be more efficient, avoiding reinventing wheels, and save us all\r\n% collectively time to solutions. Such a practical objective seems\r\n% desirable at face value. However, this first-order impact is not the only\r\n% driver for organizations pushing for Reproducibility.\r\n% \r\n%% Why is it important?\r\n% An overarching objective of the Reproducibility movement is a desire for\r\n% reliable research. In other words, research sponsors and consumers want\r\n% to know that the reported research results are accurate and have been\r\n% validated. By creating and enforcing new research and publishing best\r\n% practices around Reproducibility, sponsoring organizations, such as IEEE, \r\n% NIH, and NSF, are looking to significantly reduce future retractions,\r\n% including ones resulting from plagiarism. The hope is that if those\r\n% submitting papers for publication are required to make their research\r\n% inputs, methodologies, and outputs more transparent, the research\r\n% findings, as a result, will necessarily be more reliable. More reliable\r\n% conclusions, in turn, the theory goes, will result in fewer retractions.\r\n% As an added bonus, this reproducible research can then be reused by other\r\n% researchers.\r\n%  \r\n% This may seem like a lot of work just to avoid retractions. A shocking\r\n% fact is that there are hundreds to even thousands of retractions per\r\n% year. One 2014 <http:\/\/phys.org\/news\/2014-02-science-publisher-gibberish-papers.html article in phys.org> reported\r\n% finding 16 articles in their archives that were actually\r\n% computer-generated; yes, they were produced by a machine, not a human.\r\n% Following that, IEEE found 100 more in their archives.\r\n% \r\n%% Getting Consensus on Reproducible Research Needs and Approaches\r\n% Various organizations have held meetings on Reproducibility with the goal\r\n% of reaching some consensus on best practices and next steps.\r\n% Representatives from MathWorks have participated in multiple of these. As\r\n% examples, Brown University\u00e2\u20ac&#x2122;s ICERM held a workshop \u00e2\u20ac\u201c <https:\/\/icerm.brown.edu\/tw12-5-rcem Reproducibility in\r\n% Computational and Experimental Mathematics> \u00e2\u20ac\u201c back in 2012, and Loren\r\n% attended. And this past December, there was a <https:\/\/agu.confex.com\/agu\/fm15\/meetingapp.cgi\/Session\/8417 session dedicated to\r\n% Sustainable Software> at the American Geophysical Union (AGU) Fall Meeting\r\n% where I delivered a talk. Multiple other discussions have been held on\r\n% related topics with still more to come; it\u00e2\u20ac&#x2122;s a bold endeavor with many\r\n% players, not to mention that accessible and usable cyber-infrastructure\r\n% will need to be part of any solution that gets buy-in and use by the\r\n% target user population.\r\n% \r\n%% Individual Researcher Efforts and MATLAB Related Tools and Capabilities\r\n% Meanwhile, some researchers are forging ahead on their own, taking it\r\n% upon themselves to make their research reproducible and, at the same\r\n% time, more transparent for readers and other consumers. One such\r\n% researcher is Kevin Moerman of Trinity College and the MIT\r\n% Biomechatronics Media Lab. In a September 2015 blog post, he describes\r\n% the challenges in academia that led him to combine MATLAB and open\r\n% science.\r\n% <https:\/\/thewinnower.com\/papers\/2436-open-source-publishing-to-boost-your-career\r\n% In his post, he essentially documents a recipe> that other researchers\r\n% can follow to share code and make their work more transparent.\r\n% \r\n%%\r\n%\r\n% <<gibbonsSmall.jpg>>\r\n% \r\n% A MATLAB feature that enabled Kevin to make the toolbox even more\r\n% accessible is <https:\/\/blogs.mathworks.com\/community\/2014\/04\/18\/file-exchange-now-hosts-github\/ GitHub peering>. That means that Kevin contributed the\r\n% toolbox to <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/ MATLAB File Exchange> and set the source of the download to\r\n% GitHub, allowing him to offer two entry-points to the toolbox, with only\r\n% one set of source code.\r\n% \r\n% In his post, Kevin concludes that developing the <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/48208-gibbonREPLACE_WITH_DASH_DASHthe-geometry-and-image-based-bioengineering-add-on-for-matlab?s_tid=srchtitle Gibbon Toolbox> and\r\n% making it freely available has been a huge boost for his career.\r\n% According to Kevin, \u00e2\u20ac\u0153Publishing GIBBON open source has maximized the\r\n% impact of my work on an international level\u00e2\u20ac\ufffd.\r\n% \r\n% \r\n%\r\n%% Your Turn\r\n% How do you make your research results available to your community? Let us\r\n% know <https:\/\/blogs.mathworks.com\/loren\/?p=1349#respond here>.\r\n%\r\n%% Kevin Moerman's Affiliations \r\n% 1.  Trinity Centre for Bioengineering,\r\n% University of Dublin, Trinity College, Dublin, Ireland \r\n% \r\n% 2. Biomechatronics, Media Lab., Massachusetts Institute of Technology,\r\n% Cambridge MA, USA\r\n\r\n##### SOURCE END ##### 9597f0dbda21483f99c3be9b60097cb5\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2016\/gibbonsSmall.jpg\" onError=\"this.style.display ='none';\" \/><\/div><!--introduction--><p>Today I have a guest post from <a href=\"mailto:lisak@mathworks.com\">Lisa Kempler<\/a>, MATLAB Community Strategist, MathWorks. Lisa works with MATLAB communities in domains such as geophysics, oceanography, audiology, and more, helping users developing MATLAB based tools, and creating resources for teaching and research with MATLAB. A primary goal is to have MATLAB users share their tools and best practices more widely within their communities, enhancing their use of MATLAB, and, in turn, accelerating their work. This blog talks about some efforts in research that support that sharing objective.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2016\/02\/15\/reproducibility-musings-hey-do-that-again\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[25,76],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1349"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=1349"}],"version-history":[{"count":5,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1349\/revisions"}],"predecessor-version":[{"id":3154,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/1349\/revisions\/3154"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=1349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=1349"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=1349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}