{"id":6366,"date":"2020-08-23T17:35:46","date_gmt":"2020-08-23T21:35:46","guid":{"rendered":"https:\/\/blogs.mathworks.com\/cleve\/?p=6366"},"modified":"2020-09-25T12:33:10","modified_gmt":"2020-09-25T16:33:10","slug":"svd-quantifies-increasing-partisanship-in-the-u-s-senate","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/cleve\/2020\/08\/23\/svd-quantifies-increasing-partisanship-in-the-u-s-senate\/","title":{"rendered":"SVD Quantifies Increasing Partisanship in the U. S. Senate"},"content":{"rendered":"\r\n<div class=\"content\"><!--introduction--><p>Many observers have noted that the United States Senate has become increasingly partisan in recent years.  Votes are being made more frequently along strict political party lines.  The singular value decomposition, the SVD, of matrices derived from records of roll call votes in the senate can measure this partisanship.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#8164b194-42dd-4ba3-acd1-f87c407e530a\">The Senate<\/a><\/li><li><a href=\"#4cf96a1a-775d-4aab-9cc9-011fe27e8dca\">The Data<\/a><\/li><li><a href=\"#a7238a41-05fb-44a5-8e49-08682a76906a\">The Matrices<\/a><\/li><li><a href=\"#890c72e4-703e-4dda-b063-9b60d8a7e7c6\">The Matrices<\/a><\/li><li><a href=\"#3d7cd546-1696-4415-ae8d-11fb20975d14\">Partisanship<\/a><\/li><li><a href=\"#edb6ace2-c981-4abb-a4e1-6d546ae91fea\">Compare<\/a><\/li><li><a href=\"#bd5bd5be-c495-4c95-b1dd-74003c10556f\">Increasing Partisanship<\/a><\/li><\/ul><\/div><h4>The Senate<a name=\"8164b194-42dd-4ba3-acd1-f87c407e530a\"><\/a><\/h4><p>The web site <a href=\"https:\/\/www.senate.gov\">www.senate.gov<\/a> has year-by-year records of voting in the senate.  We are going to look at the 32 years from 1989 to 2020.<\/p><p>Senators serve six-year terms and every two years about one-third of the senate stands for reelection or new senators are elected. So, a new \"Congress\" convenes every two years and has two year-long \"Sessions\".  The first session was in 1789.  The 101-st Congress began in 1989 and this year, 2020, is the 2nd Session of the 116-th Congress.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/division.png\" alt=\"\"> <\/p><p>This bar chart shows the division between the Republican and Democratic parties over our time period. The data from the web site <a href=\"https:\/\/www.senate.gov\/history\/partydiv.htm\">party division<\/a> comes with the caveat, \"The actual number of senators representing a particular party often changes during a Congress, due to the death or resignation of a senator, or as a consequence of a member changing parties.\"<\/p><h4>The Data<a name=\"4cf96a1a-775d-4aab-9cc9-011fe27e8dca\"><\/a><\/h4><p>The URL<\/p><pre class=\"language-matlab\">https:\/\/www.senate.gov\/legislative\/LIS\/roll_call_votes\/ <span class=\"keyword\">...<\/span>\r\n   vote1162\/vote_116_2_00033.xml\r\n<\/pre><p>points to the record of a roll call vote taken in the senate early this year.  The congress number, 116, and the session, 2, each appear twice. The 33 signifies this is the 33rd vote of this session.<\/p><p>This an XML record. XML is the abbreviation for eXtensible Markup Language, an important standard from the WWW Consortium defining a format for encoding documents that is both human and machine readable. I was pleased to find MATLAB Central File Exchange <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/28518-xml2struct\">28518-xml2struct<\/a> by Wouter Falkena.  As the function name implies, this handy code \"converts an xml file into a MATLAB structure for easy access to the data.\"<\/p><p>You can also use your browser to access the example URL and (a) examine the XML, and (b) see why bills 33 and 34 from year 2020 are not typical measures before the Senate.<\/p><h4>The Matrices<a name=\"a7238a41-05fb-44a5-8e49-08682a76906a\"><\/a><\/h4><p>Once you find your way through the layers of structure that results from the  XML, you will have many details about each senate roll call vote. In particular, for each senator who voted, you will have a structure with fields full name, last name, first name, party, state, vote, and member id.<\/p><p>For each year, we create a matrix of +1s, -1s, and 0s.  A +1 is for a vote of \"Yea\" or \"Guilty\", a -1 is a \"Nay\" or \"Not Guilty\", and a 0 is for anything else, which includes \"Not Voting\", \"Present\" (abstain), and something too complicated to explain called \"Present, Giving Live Pair\".  Because there are 100 senators, the matrix has 100 rows. The number of columns is the number of roll call votes taken that year. The matrices range in size from 100-by-163 for 2016 to 100-by-613 for 1995.  The matrix for 2020 is not yet complete; as of August 6, there were 157 columns.<\/p><p>The XML is organized using last names of senators in alphabetic order -- Alexander (R-TN), Baldwin (D-WI), Barrasso (R-WY), ..., Young(R-IN). At first, I tried to create matrices with rows in this order. But senators retire or die in midsession, so this ordering proved to be difficult to manage correctly.  Ultimately, I decided to order the rows by states.  Each state is allocated two rows; the name of the specific senator occupying each row is irrelevant.  After the matrix for a given year is complete, I reorder the rows so that all the senators from each party are grouped together.  The one or two senators who declare themselves to be independent (but who caucus with a major party) are between the two parties.<\/p><h4>The Matrices<a name=\"890c72e4-703e-4dda-b063-9b60d8a7e7c6\"><\/a><\/h4><p>Here are the matrices, 32 of them, one for each session from 1989 to 2020.  Yellow is \"Yea\" or \"Guilty\", dark blue is \"Nay\" or \"Not Guilty\", and turquoise is not present, abstain or other.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/senate_small.gif\" alt=\"\"> <\/p><h4>Partisanship<a name=\"3d7cd546-1696-4415-ae8d-11fb20975d14\"><\/a><\/h4><p>The SVD of these matrices provides the basis for our analysis. The singular values decrease very rapidly with increasing index, so the matrices can be well approximated by matrices of rank two. When viewed from the record of roll call votes, the U. S. Senate is nearly two-dimensional.<\/p><p>Let <tt>A(y)<\/tt> be the matrix of 1s, -1s and 0s summarizing the roll call votes in year <tt>y<\/tt>.  Let<\/p><pre class=\"language-matlab\">sigma = svd(A(y))\r\n<\/pre><p>Then the <i>partisanship<\/i> for year <tt>y<\/tt> is defined to be<\/p><pre class=\"language-matlab\">1-sigma(3)\/sigma(1)\r\n<\/pre><p>This quantity measures how close the voting pattern is to being two-dimensional.<\/p><h4>Compare<a name=\"edb6ace2-c981-4abb-a4e1-6d546ae91fea\"><\/a><\/h4><p>Let's compare the least partisan and most partisan years between 1989 and 2020.  They turn out to be 1991 and 2014.  A good way to see the structure of these matrices is to view them as images. I have also included \"coinflip\", a matrix of random +1s and -1s where there is a complete lack of party line voting.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/A1991.png\" alt=\"\"> <\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/A2014.png\" alt=\"\"> <\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/coinflip.png\" alt=\"\"> <\/p><p>You can immediately see that 2014 has much less detailed structure than 1991; it is closer to being two-dimensional.<\/p><p>The next plot shows the distribution of the first twenty singular values, normalized so that <tt>sigma(1) = 1<\/tt>.  The third singular value is highlighted. The year 2014 is more partisan than the year 1991, and, as expected, they both show far more party line voting than random coin flips.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/compare.png\" alt=\"\"> <\/p><h4>Increasing Partisanship<a name=\"bd5bd5be-c495-4c95-b1dd-74003c10556f\"><\/a><\/h4><p>Finally, here is this measure of partisanship for each year from 1989 to 2020, and a linear least squares fit.  With this much scatter is the values, the exact fit can't be taken too seriously. But there is definitely an upward trend.  Partisanship in the U. S. Senate has increased from about 70% thirty years ago to about 80% today.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/partisanship.png\" alt=\"\"> <\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_e36681b1daef494298c33a60bb1dc16b() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='e36681b1daef494298c33a60bb1dc16b ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' e36681b1daef494298c33a60bb1dc16b';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2020 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_e36681b1daef494298c33a60bb1dc16b()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2020b<br><\/p><\/div><!--\r\ne36681b1daef494298c33a60bb1dc16b ##### SOURCE BEGIN #####\r\n%% SVD Quantifies Increasing Partisanship in the U. S. Senate\r\n% Many observers have noted that the United States Senate has become\r\n% increasingly partisan in recent years.  Votes are being made more\r\n% frequently along strict political party lines.  The singular value \r\n% decomposition, the SVD, of matrices derived from records of roll call\r\n% votes in the senate can measure this partisanship.\r\n\r\n%% The Senate\r\n% The web site <https:\/\/www.senate.gov www.senate.gov> has\r\n% year-by-year records of voting in the senate.  We are going to \r\n% look at the 32 years from 1989 to 2020.\r\n\r\n%%\r\n% Senators serve six-year terms and every two years about one-third\r\n% of the senate stands for reelection or new senators are elected.\r\n% So, a new \"Congress\" convenes every two years and has two year-long\r\n% \"Sessions\".  The first session was in 1789.  The 101-st Congress\r\n% began in 1989 and this year, 2020, is the 2nd Session of the\r\n% 116-th Congress.\r\n\r\n%%\r\n%\r\n% <<division.png>>\r\n%\r\n% This bar chart shows the division between the\r\n% Republican and Democratic parties over our time period.\r\n% The data from the web site <https:\/\/www.senate.gov\/history\/partydiv.htm\r\n% party division> comes with the caveat,\r\n% \"The actual number of senators representing a particular party often\r\n% changes during a Congress, due to the death or resignation of a senator,\r\n% or as a consequence of a member changing parties.\"\r\n\r\n%% The Data\r\n% The URL\r\n%\r\n%   https:\/\/www.senate.gov\/legislative\/LIS\/roll_call_votes\/ ...\r\n%      vote1162\/vote_116_2_00033.xml\r\n%\r\n% points to the record of a roll call vote taken in the senate early this\r\n% year.  The congress number, 116, and the session, 2, each appear twice. \r\n% The 33 signifies this is the 33rd vote of this session.\r\n\r\n%%\r\n% This an XML record. \r\n% XML is the abbreviation for eXtensible Markup Language, an important\r\n% standard from the WWW Consortium defining a format for encoding\r\n% documents that is both human and machine readable.\r\n% I was pleased to find MATLAB Central File Exchange\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/28518-xml2struct\r\n% 28518-xml2struct> by Wouter Falkena.  As the function name implies,\r\n% this handy code \"converts an xml file into a MATLAB structure for\r\n% easy access to the data.\"\r\n\r\n%%\r\n% You can also use your browser to access the example URL and (a) examine \r\n% the XML, and (b) see why bills 33 and 34 from year 2020 are not typical\r\n% measures before the Senate.\r\n\r\n%% The Matrices\r\n% Once you find your way through the layers of structure that results from\r\n% the  XML, you will have many details about each senate roll call vote.\r\n% In particular, for each senator who voted, you will have a structure\r\n% with fields full name, last name, first name, party, state, vote,\r\n% and member id.\r\n\r\n%% \r\n% For each year, we create a matrix of +1s, -1s, and 0s.  A +1 is for\r\n% a vote of \"Yea\" or \"Guilty\", a -1 is a \"Nay\" or \"Not Guilty\", and a \r\n% 0 is for anything else, which includes \"Not Voting\", \"Present\" (abstain),\r\n% and something too complicated to explain called \"Present, Giving Live\r\n% Pair\".  Because there are 100 senators, the matrix has 100 rows. \r\n% The number of columns is the number of roll call votes taken that year.  \r\n% The matrices range in size from 100-by-163 for 2016 to \r\n% 100-by-613 for 1995.  The matrix for 2020 is not yet complete;\r\n% as of August 6, there were 157 columns.\r\n\r\n%%\r\n% The XML is organized using last names of senators in alphabetic order REPLACE_WITH_DASH_DASH\r\n% Alexander (R-TN), Baldwin (D-WI), Barrasso (R-WY), ..., Young(R-IN).\r\n% At first, I tried to create matrices with rows in this order.   \r\n% But senators retire or die in midsession, so this ordering proved to be\r\n% difficult to manage correctly.  Ultimately, I decided to order the rows\r\n% by states.  Each state is allocated two rows; the name of the specific\r\n% senator occupying each row is irrelevant.  After the matrix for\r\n% a given year is complete, I reorder the rows so that all the senators\r\n% from each party are grouped together.  The one or two senators who\r\n% declare themselves to be independent (but who caucus with a major party)\r\n% are between the two parties.\r\n\r\n%% The Matrices\r\n% Here are the matrices, 32 of them, one for each session from 1989 to\r\n% 2020.  Yellow is \"Yea\" or \"Guilty\", dark blue is \"Nay\" or \"Not Guilty\",\r\n% and turquoise is not present, abstain or other.\r\n%\r\n% <<senate_small.gif>>\r\n\r\n%% Partisanship\r\n% The SVD of these matrices provides the basis for our analysis.\r\n% The singular values decrease very rapidly with increasing index,\r\n% so the matrices can be well approximated by matrices of rank two.\r\n% When viewed from the record of roll call votes, the U. S. Senate\r\n% is nearly two-dimensional.\r\n\r\n%%\r\n% Let |A(y)| be the matrix of 1s, -1s and 0s summarizing the roll call\r\n% votes in year |y|.  Let\r\n%\r\n%   sigma = svd(A(y))\r\n%\r\n% Then the _partisanship_ for year |y| is defined to be\r\n%\r\n%   1-sigma(3)\/sigma(1) \r\n%\r\n% This quantity measures how close the voting pattern is to\r\n% being two-dimensional.\r\n\r\n%% Compare\r\n% Let's compare the least partisan and most partisan years between 1989\r\n% and 2020.  They turn out to be 1991 and 2014.  A good way to see the\r\n% structure of these matrices is to view them as images. \r\n% I have also included \"coinflip\", a matrix of random +1s and -1s where\r\n% there is a complete lack of party line voting.\r\n%%\r\n%\r\n% <<A1991.png>>\r\n%\r\n% <<A2014.png>>\r\n%\r\n% <<coinflip.png>>\r\n%\r\n\r\n%%\r\n% You can immediately see that 2014 has much less detailed structure than\r\n% 1991; it is closer to being two-dimensional.\r\n%\r\n% The next plot shows the distribution of the first twenty singular values,\r\n% normalized so that |sigma(1) = 1|.  The third singular value is\r\n% highlighted. The year 2014 is more partisan than the year 1991, and,\r\n% as expected, they both show far more party line voting than random\r\n% coin flips.\r\n%\r\n% <<compare.png>>\r\n%\r\n\r\n%% Increasing Partisanship\r\n% Finally, here is this measure of partisanship for each year from 1989\r\n% to 2020, and a linear least squares fit.  With this much\r\n% scatter is the values, the exact fit can't be taken too seriously.\r\n% But there is definitely an upward trend.  Partisanship in the U. S. Senate\r\n% has increased from about 70% thirty years ago to about 80% today.\r\n%\r\n% <<partisanship.png>>\r\n%\r\n% \r\n##### SOURCE END ##### e36681b1daef494298c33a60bb1dc16b\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/partisanship.png\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" decoding=\"async\" loading=\"lazy\" \/><\/div><!--introduction--><p>Many observers have noted that the United States Senate has become increasingly partisan in recent years.  Votes are being made more frequently along strict political party lines.  The singular value decomposition, the SVD, of matrices derived from records of roll call votes in the senate can measure this partisanship.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/cleve\/2020\/08\/23\/svd-quantifies-increasing-partisanship-in-the-u-s-senate\/\">read more >><\/a><\/p>","protected":false},"author":78,"featured_media":6380,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,8,30],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/6366"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/comments?post=6366"}],"version-history":[{"count":4,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/6366\/revisions"}],"predecessor-version":[{"id":6401,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/6366\/revisions\/6401"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/media\/6380"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/media?parent=6366"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/categories?post=6366"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/tags?post=6366"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}