{"id":892,"date":"2014-05-07T09:52:10","date_gmt":"2014-05-07T14:52:10","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=892"},"modified":"2015-08-27T14:14:47","modified_gmt":"2015-08-27T19:14:47","slug":"long-term-analysis-of-the-sp-500","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2014\/05\/07\/long-term-analysis-of-the-sp-500\/","title":{"rendered":"Long-term Analysis of the S&#038;P 500"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>Here at the MathWorks we have a great staff to support the financial industry. We also have our fair share of DIYers who like to follow the market and manage their own investments, such as this edition's guest blogger, Steve Cruickshank, an admitted \"cocktail party economist\".<\/p><p>During his day job, Steve is the marketing manager for our MATLAB <a href=\"https:\/\/www.mathworks.com\/solutions\/desktop-web-deployment.html\">Application Deployment<\/a> products.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#ffd9b715-631d-4d4d-9af3-0af3962c829b\">Introduction - Long term tracking and statistical analysis of the S&amp;P 500<\/a><\/li><li><a href=\"#c05c089e-dd1d-425f-b6c6-bae2e71c9101\">Setting the initial constraints<\/a><\/li><li><a href=\"#3d3e6f99-3fbb-42c0-ae80-0fbc88a152b7\">Curve fit calculations<\/a><\/li><li><a href=\"#787eef34-ed6f-40f0-b2ae-a2f98cd4111d\">Plotting and visualizing<\/a><\/li><li><a href=\"#069aa602-1c7f-44f9-9fa4-49d577c58762\">Quantifying the data with statistics<\/a><\/li><li><a href=\"#653a42e6-b6b7-4125-bd8f-f974c99a6f37\">Conclusions<\/a><\/li><\/ul><\/div><h4>Introduction - Long term tracking and statistical analysis of the S&amp;P 500<a name=\"ffd9b715-631d-4d4d-9af3-0af3962c829b\"><\/a><\/h4><p>If you&#8217;re an amateur investor like me, you probably experienced great frustration during late 2008 and early 2009, watching the value of your various stock portfolios, IRAs, and 401K accounts plummet. Sure, we lived the high life during the run-ups of the late 90s dot-com boom and subsequent gains during 2003 &#8211; 2008, but the Great Recession of late 2008 and early 2009 made me take a closer look at the S&amp;P 500, especially the long term trends.<\/p><p>Caveat emptor - I&#8217;m not a financial industry professional and couldn&#8217;t tell you the difference between GARCH and Black-Scholes without using Wikipedia, but I do enjoy managing my own investments and crunching numbers to gain perspective. As long as we&#8217;re in &#8220;full disclosure&#8221; mode, I borrowed the phrase &#8220;cocktail party economist&#8221; from my grad school Economics professor.<\/p><p>When looking at the S&amp;P 500 trends over various timeframes, or reading web pages from the various pundits, it&#8217;s easy to see that you can cherry pick a time period to validate virtually any hypothesis. Think the market is below a reasonable level? Then start your analysis fifteen years ago &#8230; 1999 &#8230; the height of the dot com boom. Since then the S&amp;P has grown a paltry 1.4% annually, far below the historical average annual growth rate of ~7.5%. Plenty of room for growth, so buy everything in sight.<\/p><p>Want to espouse the theory that the market has grown way too fast and is in yet another bubble just waiting to burst? Sure, we can do that! Start your analysis five years ago &#8230; early 2009 &#8230; the bottom of the financial crisis. Since then the S&amp;P has grown at a ridiculous rate, over 20% annually, roughly the same as during the dot com boom years. Time to sell everything.<\/p><p>Given those obvious extremes, perhaps the best way to eliminate the false conclusions reached by cherry picking time periods is to analyze data from as far back as possible and evaluate the current market based on the long term trends. The example uses Yahoo Finance to provide data for the S&amp;P 500 since 1950, but you could use other data sources and time frames as you see fit.<\/p><p>The MathWorks' software used in this example is <a href=\"https:\/\/www.mathworks.com\/products\/matlab\/\">MATLAB<\/a> and the <a href=\"https:\/\/www.mathworks.com\/products\/datafeed\/\">Datafeed Toolbox<\/a>. Home users may want to use <a href=\"https:\/\/www.mathworks.com\/products\/matlab-home\/\">MATLAB Home<\/a> and hard-core number crunchers may prefer the advanced functionality provided by the <a href=\"https:\/\/www.mathworks.com\/products\/curvefitting\/\">Curve Fitting Toolbox<\/a>.<\/p><h4>Setting the initial constraints<a name=\"c05c089e-dd1d-425f-b6c6-bae2e71c9101\"><\/a><\/h4><p>I am using <a title=\"https:\/\/www.mathworks.com\/help\/finance\/fetch.html (link no longer works)\"><tt>fetch<\/tt><\/a> from the <a href=\"https:\/\/www.mathworks.com\/products\/datafeed\/\">Datafeed Toolbox<\/a> to pull data from Yahoo based on a user specified timeframe, i.e. how many years from now. This example uses the 'w' option to get <i>weekly<\/i> data, but could also use 'm' for <i>month<\/i> or 'd' for <i>day<\/i> without materially affecting the results and conclusions.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Set Yahoo as data source.<\/span>\r\ny = yahoo;\r\n\r\n<span class=\"comment\">% Determine how far back to look. Yahoo provides data starting at 1950,<\/span>\r\n<span class=\"comment\">% so 65 years gathers all the available data.<\/span>\r\nyears = 65;\r\nstart = now - (365.25 * years);\r\n\r\n<span class=\"comment\">% Get data from Yahoo.<\/span>\r\nCloseData = fetch(y, <span class=\"string\">'^GSPC'<\/span>, <span class=\"string\">'Close'<\/span>, start, now, <span class=\"string\">'w'<\/span>);\r\n<\/pre><h4>Curve fit calculations<a name=\"3d3e6f99-3fbb-42c0-ae80-0fbc88a152b7\"><\/a><\/h4><p>We can then extract the closing price and date for each point, using the <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/polyfit.html\"><tt>polyfit<\/tt><\/a> and <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/polyval.html\"><tt>polyval<\/tt><\/a> functions to determine the best fit line through the data.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Extract data of interest from Yahoo results<\/span>\r\nClosePrice = CloseData(:,2);\r\nCloseDate = CloseData(:,1);\r\n\r\n<span class=\"comment\">% Curve fit using a polynomial fit to create curve fit item and trend data<\/span>\r\n<span class=\"comment\">% based on curve fit result.<\/span>\r\nTrendLine = polyfit(CloseDate,log(ClosePrice), 1);\r\nTrendData = polyval(TrendLine,CloseDate);\r\n<\/pre><p>Option: If your analysis requires other types of curve fitting, you could also use <a href=\"https:\/\/www.mathworks.com\/products\/curvefitting\/\">Curve Fitting Toolbox<\/a>, as shown here:<\/p><pre>  TrendLine2 = fit(CloseDate, log(ClosePrice), 'poly1');\r\n  TrendData = feval(TrendLine2, CloseDate);<\/pre><h4>Plotting and visualizing<a name=\"787eef34-ed6f-40f0-b2ae-a2f98cd4111d\"><\/a><\/h4><p>Plotting the variables <tt>CloseDate<\/tt>, <tt>ClosePrice<\/tt>, and <tt>TrendData<\/tt> on a log plot provides the following image. The variables plotted along the y-axis (<tt>ClosePrice<\/tt> and <tt>TrendData<\/tt>) are expressed as exponential functions, such as $ClosePrice = e^y$. Plotting data using exponents and\/or log plots provides a better visual representation of average growth rates over long periods of time.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Plotting the actual performance against best fit line<\/span>\r\nplot(CloseDate,log(ClosePrice),CloseDate,TrendData);\r\ntitle([<span class=\"string\">'S&amp;P 500 Performance and Best Fit Curve, last '<\/span>,num2str(years),<span class=\"string\">' years'<\/span>])\r\nylabel(<span class=\"string\">'Closing and Best Fit Prices (e^y)'<\/span>)\r\nxlabel(<span class=\"string\">'Closing Dates'<\/span>)\r\ndatetick(<span class=\"string\">'x'<\/span>);\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2014\/SP500_StdDev_Analysis_29Apr14_01.png\" alt=\"\"> <p>This plot provides an effective visual mechanism to tell if the market is currently above or below the long term average. Interestingly, the current market (as of late-April 2014) is landing very close to the best fit curve, so perhaps the market is just about where it &#8220;should&#8221; be. You probably won&#8217;t hear that sentiment from market prognosticators because bulls, bears, and bubbles generate more page views.<\/p><p>Using the <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/axis.html\"><tt>axis<\/tt><\/a> function to zoom in on the previous plot, we can highlight two easily recognizable data points: the peak of the late &#8216;90&#8217;s dot com bubble and the financial crisis roughly ten years later. In this example, the <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/annotation.html\"><tt>annotation<\/tt><\/a> function provides the graphics.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Setting appropriate zoom coordinates<\/span>\r\naxis([729661, 736053, 6.4, 7.6])\r\n\r\n<span class=\"comment\">% Create ellipses to highlight areas of interest<\/span>\r\nannotation(<span class=\"string\">'ellipse'<\/span>,[0.168, 0.652, 0.149, 0.152], <span class=\"string\">'LineWidth'<\/span>,2, <span class=\"string\">'Color'<\/span>,<span class=\"string\">'r'<\/span>);\r\nannotation(<span class=\"string\">'ellipse'<\/span>,[0.592, 0.183, 0.078, 0.431], <span class=\"string\">'LineWidth'<\/span>,2, <span class=\"string\">'Color'<\/span>,<span class=\"string\">'r'<\/span>);\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2014\/SP500_StdDev_Analysis_29Apr14_02.png\" alt=\"\"> <h4>Quantifying the data with statistics<a name=\"069aa602-1c7f-44f9-9fa4-49d577c58762\"><\/a><\/h4><p>Visuals are nice, but what if you, like so many of us here at MathWorks, prefer numbers and want to quantify the information rather than eyeball it? The first step in quantifying this data is to determine the variance, the variable <tt>Delta<\/tt> in my example, between the actual closing price and the projected price based on the best fit line.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Calculate the difference between actual results and best fit curve<\/span>\r\nDelta = log(ClosePrice) - TrendData;\r\n<\/pre><p>The calculations above will transpose the sloped line of the best fit curve to a horizontal line, but the familiar data points from the previous figure are clear, the dot com boom and financial crisis. We can then plot <tt>Delta<\/tt> relative to the best fit curve using the functions below. The <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/zeros.html\"><tt>zeros<\/tt><\/a> function converts a single '0' to a vector for effective plotting.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Clear the figure then plot the raw data. Note that the BestFitCurve is<\/span>\r\n<span class=\"comment\">% multiplied by zeros to get reasonable legend<\/span>\r\nclf\r\nh1 = plot(CloseDate, Delta);\r\ntitle([<span class=\"string\">'S&amp;P 500 Variance from Best Fit Curve, last '<\/span>,num2str(years),<span class=\"string\">' years'<\/span>])\r\ndatetick (<span class=\"string\">'x'<\/span>);\r\nBestFitCurve = line(CloseDate, zeros(size(CloseDate)), <span class=\"string\">'Color'<\/span>, <span class=\"string\">'k'<\/span>);\r\nylabel(<span class=\"string\">'Variance from Best Fit Line'<\/span>)\r\nxlabel(<span class=\"string\">'Closing Dates'<\/span>)\r\n\r\n<span class=\"comment\">% Create ellipses to highlight areas of interest<\/span>\r\nannotation(<span class=\"string\">'ellipse'<\/span>, [0.753, 0.155, 0.077, 0.364], <span class=\"string\">'LineWidth'<\/span>,2, <span class=\"string\">'Color'<\/span>,<span class=\"string\">'r'<\/span>);\r\nannotation(<span class=\"string\">'ellipse'<\/span>, [0.631, 0.793, 0.104, 0.131], <span class=\"string\">'LineWidth'<\/span>,2, <span class=\"string\">'Color'<\/span>,<span class=\"string\">'r'<\/span>);\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2014\/SP500_StdDev_Analysis_29Apr14_03.png\" alt=\"\"> <p>To continue quantifying the results, we can use the <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/std.html\"><tt>std<\/tt><\/a> function to determine the standard deviation, variable name <tt>Sigma<\/tt>, for the variable <tt>Delta<\/tt>. To cover a broader range of variances we can then apply multipliers (2, 1, &frac12;, -&frac12;, -1, -2) to <tt>Sigma<\/tt>, showing statistically how far above or below the market is with respect to the long term average. Thankfully none of the market fluctuations reached the Six Sigma levels.<\/p><pre class=\"codeinput\"><span class=\"comment\">% Define Sigma as the standard deviation of Delta, then convert to a vector<\/span>\r\n<span class=\"comment\">% using a matrix of ones<\/span>\r\nSigma = std(Delta)*ones(size(CloseDate));\r\n\r\n<span class=\"comment\">% Clear the previous figure, define the various Sigma's, then plot the data<\/span>\r\n<span class=\"comment\">% with a legend<\/span>\r\nclf\r\nh1 = plot (CloseDate, Delta);\r\ntitle([<span class=\"string\">'S&amp;P 500 Variance from Best Fit Curve, last '<\/span>,num2str(years),<span class=\"string\">' years'<\/span>])\r\ndatetick(<span class=\"string\">'x'<\/span>);\r\nBestFitCurve = line(CloseDate, zeros(size(CloseDate)), <span class=\"string\">'Color'<\/span>, <span class=\"string\">'k'<\/span>);\r\nTwoSigma = line(CloseDate, 2*Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'g'<\/span>, <span class=\"string\">'LineStyle'<\/span>, <span class=\"string\">':'<\/span>);\r\nOneSigma = line(CloseDate, Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'g'<\/span>, <span class=\"string\">'LineStyle'<\/span>, <span class=\"string\">'--'<\/span>);\r\nHalfSigma = line(CloseDate, 0.5*Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'g'<\/span>);\r\nNegHalfSigma = line(CloseDate, -0.5*Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'r'<\/span>);\r\nNegOneSigma = line(CloseDate, -Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'r'<\/span>, <span class=\"string\">'LineStyle'<\/span>, <span class=\"string\">'--'<\/span>);\r\nNegTwoSigma = line(CloseDate, -2*Sigma, <span class=\"string\">'Color'<\/span>, <span class=\"string\">'r'<\/span>, <span class=\"string\">'LineStyle'<\/span>, <span class=\"string\">':'<\/span>);\r\nylabel(<span class=\"string\">'Variance from Best Fit Line'<\/span>)\r\nxlabel(<span class=\"string\">'Closing Dates'<\/span>)\r\n\r\nlegend([h1 TwoSigma OneSigma HalfSigma BestFitCurve <span class=\"keyword\">...<\/span>\r\n    NegHalfSigma NegOneSigma NegTwoSigma], <span class=\"keyword\">...<\/span>\r\n    {<span class=\"string\">'Delta'<\/span>, <span class=\"string\">'+ 2\\sigma'<\/span>, <span class=\"string\">'+ \\sigma'<\/span>, <span class=\"string\">'+ \\sigma\/2'<\/span> <span class=\"string\">'Best Fit'<\/span>, <span class=\"keyword\">...<\/span>\r\n    <span class=\"string\">'- \\sigma\/2'<\/span>, <span class=\"string\">'- \\sigma'<\/span>,<span class=\"string\">'- 2\\sigma'<\/span>}, <span class=\"string\">'Location'<\/span>, <span class=\"string\">'NorthEastOutside'<\/span>)\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2014\/SP500_StdDev_Analysis_29Apr14_04.png\" alt=\"\"> <h4>Conclusions<a name=\"653a42e6-b6b7-4125-bd8f-f974c99a6f37\"><\/a><\/h4><p>With data and analysis like above, hindsight is always 20\/20. In a perfect world, most investors like myself probably wish they had sold everything when the S&amp;P went above the +2Sigma level near the end of the dot com bubble, and bought as much as possible on margin during extremely short window below -2Sigma in early 2009.<\/p><p>We all have our own risk profiles, so I won&#8217;t pretend to tell you what level represents an attractive buying or selling opportunity ... you need to do your own homework and decide for yourself. The example in this article is just one method to cut through the propaganda and utilize math to provide relatively unbiased data.<\/p><p>One final question for the readers ... do you agree with the statement made earlier that the current level of S&amp;P 500 is about where it \"should\" be given the long term data? If not, what's your expected level and rationale?  Please let me know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=892#respond\">here<\/a>.<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_0c203620ea8344b98a0d5ebb76dc3402() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='0c203620ea8344b98a0d5ebb76dc3402 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 0c203620ea8344b98a0d5ebb76dc3402';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2014 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_0c203620ea8344b98a0d5ebb76dc3402()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2014a<br><\/p><\/div><!--\r\n0c203620ea8344b98a0d5ebb76dc3402 ##### SOURCE BEGIN #####\r\n%% Long-term Analysis of the S&P 500\r\n% Here at the MathWorks we have a great staff to support the financial\r\n% industry. We also have our fair share of DIYers who like to follow the\r\n% market and manage their own investments, such as this edition's guest\r\n% blogger, Steve Cruickshank, an admitted \"cocktail party economist\".\r\n%\r\n% During his day job, Steve is the marketing manager for our MATLAB\r\n% <https:\/\/www.mathworks.com\/solutions\/desktop-web-deployment.html Application Deployment> products.\r\n\r\n%% Introduction - Long term tracking and statistical analysis of the S&P 500\r\n% If you\u00e2\u20ac\u2122re an amateur investor like me, you probably experienced great\r\n% frustration during late 2008 and early 2009, watching the value of your\r\n% various stock portfolios, IRAs, and 401K accounts plummet. Sure, we\r\n% lived the high life during the run-ups of the late 90s dot-com boom and\r\n% subsequent gains during 2003 \u00e2\u20ac\u201c 2008, but the Great Recession of late 2008\r\n% and early 2009 made me take a closer look at the S&P 500, especially the\r\n% long term trends.\r\n%\r\n% Caveat emptor - I\u00e2\u20ac\u2122m not a financial industry professional and couldn\u00e2\u20ac\u2122t\r\n% tell you the difference between GARCH and Black-Scholes without using\r\n% Wikipedia, but I do enjoy managing my own investments and crunching\r\n% numbers to gain perspective. As long as we\u00e2\u20ac\u2122re in \u00e2\u20ac\u0153full disclosure\u00e2\u20ac\ufffd mode,\r\n% I borrowed the phrase \u00e2\u20ac\u0153cocktail party economist\u00e2\u20ac\ufffd from my grad school\r\n% Economics professor.\r\n%\r\n% When looking at the S&P 500 trends over various timeframes, or reading\r\n% web pages from the various pundits, it\u00e2\u20ac\u2122s easy to see that you can cherry\r\n% pick a time period to validate virtually any hypothesis. Think the market\r\n% is below a reasonable level? Then start your analysis fifteen years ago \u00e2\u20ac\u00a6\r\n% 1999 \u00e2\u20ac\u00a6 the height of the dot com boom. Since then the S&P has grown a\r\n% paltry 1.4% annually, far below the historical average annual growth rate\r\n% of ~7.5%. Plenty of room for growth, so buy everything in sight.\r\n%\r\n% Want to espouse the theory that the market has grown way too fast and is\r\n% in yet another bubble just waiting to burst? Sure, we can do that! Start\r\n% your analysis five years ago \u00e2\u20ac\u00a6 early 2009 \u00e2\u20ac\u00a6 the bottom of the financial\r\n% crisis. Since then the S&P has grown at a ridiculous rate, over 20%\r\n% annually, roughly the same as during the dot com boom years. Time to sell\r\n% everything.\r\n%\r\n% Given those obvious extremes, perhaps the best way to eliminate the false\r\n% conclusions reached by cherry picking time periods is to analyze data\r\n% from as far back as possible and evaluate the current market based on the\r\n% long term trends. The example uses Yahoo Finance to provide data for\r\n% the S&P 500 since 1950, but you could use other data sources and time\r\n% frames as you see fit.\r\n%\r\n% The MathWorks' software used in this example is\r\n% <https:\/\/www.mathworks.com\/products\/matlab\/ MATLAB> and the\r\n% <https:\/\/www.mathworks.com\/products\/datafeed\/ Datafeed Toolbox>. Home\r\n% users may want to use <https:\/\/www.mathworks.com\/products\/matlab-home\/\r\n% MATLAB Home> and hard-core number crunchers may prefer the advanced\r\n% functionality provided by the\r\n% <https:\/\/www.mathworks.com\/products\/curvefitting\/ Curve Fitting Toolbox>.\r\n\r\n\r\n\r\n%% Setting the initial constraints\r\n% I am using <https:\/\/www.mathworks.com\/help\/finance\/fetch.html\r\n% |fetch|> from the <https:\/\/www.mathworks.com\/products\/datafeed\/ Datafeed\r\n% Toolbox> to pull data from Yahoo based on a user specified timeframe,\r\n% i.e. how many years from now. This example uses the 'w' option to get\r\n% _weekly_ data, but could also use 'm' for _month_ or 'd' for _day_\r\n% without materially affecting the results and conclusions.\r\n\r\n% Set Yahoo as data source.\r\ny = yahoo;\r\n\r\n% Determine how far back to look. Yahoo provides data starting at 1950,\r\n% so 65 years gathers all the available data.\r\nyears = 65;\r\nstart = now - (365.25 * years);\r\n\r\n% Get data from Yahoo. \r\nCloseData = fetch(y, '^GSPC', 'Close', start, now, 'w');\r\n\r\n%% Curve fit calculations\r\n% We can then extract the closing price and date for each point, using the\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/polyfit.html |polyfit|> and\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/polyval.html |polyval|>\r\n% functions to determine the best fit line through the data.\r\n\r\n% Extract data of interest from Yahoo results\r\nClosePrice = CloseData(:,2);\r\nCloseDate = CloseData(:,1);\r\n\r\n% Curve fit using a polynomial fit to create curve fit item and trend data\r\n% based on curve fit result.\r\nTrendLine = polyfit(CloseDate,log(ClosePrice), 1);\r\nTrendData = polyval(TrendLine,CloseDate);\r\n\r\n%%\r\n% Option: If your analysis requires other types of curve fitting, you could\r\n% also use <https:\/\/www.mathworks.com\/products\/curvefitting\/ Curve Fitting\r\n% Toolbox>, as shown here:\r\n%\r\n%    TrendLine2 = fit(CloseDate, log(ClosePrice), 'poly1'); \r\n%    TrendData = feval(TrendLine2, CloseDate);\r\n\r\n\r\n%% Plotting and visualizing\r\n% Plotting the variables |CloseDate|, |ClosePrice|, and |TrendData| on a log plot\r\n% provides the following image. The variables plotted along the y-axis\r\n% (|ClosePrice| and |TrendData|) are expressed as exponential functions, such\r\n% as $ClosePrice = e^y$. Plotting data using exponents and\/or log plots\r\n% provides a better visual representation of average growth rates over long\r\n% periods of time.\r\n\r\n% Plotting the actual performance against best fit line\r\nplot(CloseDate,log(ClosePrice),CloseDate,TrendData);\r\ntitle(['S&P 500 Performance and Best Fit Curve, last ',num2str(years),' years'])\r\nylabel('Closing and Best Fit Prices (e^y)')\r\nxlabel('Closing Dates')\r\ndatetick('x');\r\n\r\n%%\r\n% This plot provides an effective visual mechanism to tell if the\r\n% market is currently above or below the long term average. Interestingly,\r\n% the current market (as of late-April 2014) is landing very close to the\r\n% best fit curve, so perhaps the market is just about where it \u00e2\u20ac\u0153should\u00e2\u20ac\ufffd be.\r\n% You probably won\u00e2\u20ac\u2122t hear that sentiment from market prognosticators\r\n% because bulls, bears, and bubbles generate more page views.\r\n%\r\n% Using the <https:\/\/www.mathworks.com\/help\/matlab\/ref\/axis.html |axis|>\r\n% function to zoom in on the previous plot, we can highlight two easily\r\n% recognizable data points: the peak of the late \u00e2\u20ac\u02dc90\u00e2\u20ac\u2122s dot com bubble and\r\n% the financial crisis roughly ten years later. In this example, the\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/annotation.html |annotation|>\r\n% function provides the graphics.\r\n\r\n% Setting appropriate zoom coordinates\r\naxis([729661, 736053, 6.4, 7.6])\r\n\r\n% Create ellipses to highlight areas of interest\r\nannotation('ellipse',[0.168, 0.652, 0.149, 0.152], 'LineWidth',2, 'Color','r');\r\nannotation('ellipse',[0.592, 0.183, 0.078, 0.431], 'LineWidth',2, 'Color','r');\r\n\r\n%% Quantifying the data with statistics\r\n% Visuals are nice, but what if you, like so many of us here at\r\n% MathWorks, prefer numbers and want to quantify the information rather\r\n% than eyeball it? The first step in quantifying this data is to determine\r\n% the variance, the variable |Delta| in my example, between the actual\r\n% closing price and the projected price based on the best fit line.\r\n\r\n% Calculate the difference between actual results and best fit curve\r\nDelta = log(ClosePrice) - TrendData;\r\n\r\n%%\r\n% The calculations above will transpose the sloped line of the best fit\r\n% curve to a horizontal line, but the familiar data points from the\r\n% previous figure are clear, the dot com boom and financial crisis. We can\r\n% then plot |Delta| relative to the best fit curve using the functions below.\r\n% The <https:\/\/www.mathworks.com\/help\/matlab\/ref\/zeros.html |zeros|> function\r\n% converts a single '0' to a vector for effective plotting.\r\n% \r\n\r\n% Clear the figure then plot the raw data. Note that the BestFitCurve is\r\n% multiplied by zeros to get reasonable legend\r\nclf\r\nh1 = plot(CloseDate, Delta);\r\ntitle(['S&P 500 Variance from Best Fit Curve, last ',num2str(years),' years'])\r\ndatetick ('x');\r\nBestFitCurve = line(CloseDate, zeros(size(CloseDate)), 'Color', 'k');\r\nylabel('Variance from Best Fit Line')\r\nxlabel('Closing Dates')\r\n\r\n% Create ellipses to highlight areas of interest\r\nannotation('ellipse', [0.753, 0.155, 0.077, 0.364], 'LineWidth',2, 'Color','r');\r\nannotation('ellipse', [0.631, 0.793, 0.104, 0.131], 'LineWidth',2, 'Color','r');\r\n\r\n\r\n%%\r\n% To continue quantifying the results, we can use the\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/std.html |std|> function to\r\n% determine the standard deviation, variable name |Sigma|, for the variable\r\n% |Delta|. To cover a broader range of variances we can then apply\r\n% multipliers (2, 1, \u00c2\u00bd, -\u00c2\u00bd, -1, -2) to |Sigma|, showing statistically how\r\n% far above or below the market is with respect to the long term average.\r\n% Thankfully none of the market fluctuations reached the Six Sigma levels.\r\n\r\n% Define Sigma as the standard deviation of Delta, then convert to a vector\r\n% using a matrix of ones\r\nSigma = std(Delta)*ones(size(CloseDate));\r\n\r\n% Clear the previous figure, define the various Sigma's, then plot the data\r\n% with a legend\r\nclf\r\nh1 = plot (CloseDate, Delta);\r\ntitle(['S&P 500 Variance from Best Fit Curve, last ',num2str(years),' years'])\r\ndatetick('x');\r\nBestFitCurve = line(CloseDate, zeros(size(CloseDate)), 'Color', 'k');\r\nTwoSigma = line(CloseDate, 2*Sigma, 'Color', 'g', 'LineStyle', ':');\r\nOneSigma = line(CloseDate, Sigma, 'Color', 'g', 'LineStyle', 'REPLACE_WITH_DASH_DASH');\r\nHalfSigma = line(CloseDate, 0.5*Sigma, 'Color', 'g');\r\nNegHalfSigma = line(CloseDate, -0.5*Sigma, 'Color', 'r');\r\nNegOneSigma = line(CloseDate, -Sigma, 'Color', 'r', 'LineStyle', 'REPLACE_WITH_DASH_DASH');\r\nNegTwoSigma = line(CloseDate, -2*Sigma, 'Color', 'r', 'LineStyle', ':');\r\nylabel('Variance from Best Fit Line')\r\nxlabel('Closing Dates')\r\n\r\nlegend([h1 TwoSigma OneSigma HalfSigma BestFitCurve ...\r\n    NegHalfSigma NegOneSigma NegTwoSigma], ...\r\n    {'Delta', '+ 2\\sigma', '+ \\sigma', '+ \\sigma\/2' 'Best Fit', ...\r\n    '- \\sigma\/2', '- \\sigma','- 2\\sigma'}, 'Location', 'NorthEastOutside')\r\n\r\n%% Conclusions\r\n% With data and analysis like above, hindsight is always 20\/20. In a\r\n% perfect world, most investors like myself probably wish they had sold\r\n% everything when the S&P went above the +2Sigma level near the end of the\r\n% dot com bubble, and bought as much as possible on margin during extremely\r\n% short window below -2Sigma in early 2009.\r\n%\r\n% We all have our own risk profiles, so I won\u00e2\u20ac\u2122t pretend to tell you what\r\n% level represents an attractive buying or selling opportunity ... you need\r\n% to do your own homework and decide for yourself. The example in this\r\n% article is just one method to cut through the propaganda and utilize math\r\n% to provide relatively unbiased data.\r\n%\r\n% One final question for the readers ... do you agree with the statement\r\n% made earlier that the current level of S&P 500 is about where it \"should\"\r\n% be given the long term data? If not, what's your expected level and\r\n% rationale?  Please let me know\r\n% <https:\/\/blogs.mathworks.com\/loren\/?p=*92#respond here>.\r\n\r\n\r\n##### SOURCE END ##### 0c203620ea8344b98a0d5ebb76dc3402\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/2014\/SP500_StdDev_Analysis_29Apr14_04.png\" onError=\"this.style.display ='none';\" \/><\/div><!--introduction--><p>Here at the MathWorks we have a great staff to support the financial industry. We also have our fair share of DIYers who like to follow the market and manage their own investments, such as this edition's guest blogger, Steve Cruickshank, an admitted \"cocktail party economist\".... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2014\/05\/07\/long-term-analysis-of-the-sp-500\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[33],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/892"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=892"}],"version-history":[{"count":5,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/892\/revisions"}],"predecessor-version":[{"id":897,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/892\/revisions\/897"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=892"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=892"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=892"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}