{"id":4410,"date":"2019-01-18T12:48:32","date_gmt":"2019-01-18T17:48:32","guid":{"rendered":"https:\/\/blogs.mathworks.com\/cleve\/?p=4410"},"modified":"2019-01-18T12:48:32","modified_gmt":"2019-01-18T17:48:32","slug":"floating-point-arithmetic-before-ieee-754","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/cleve\/2019\/01\/18\/floating-point-arithmetic-before-ieee-754\/","title":{"rendered":"Floating Point Arithmetic Before IEEE 754"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>In a comment following <a href=\"https:\/\/blogs.mathworks.com\/cleve\/2019\/01\/16\/variable-format-half-precision-floating-point-arithmetic\/\">my post about half-precision arithmetic<\/a>, \"Raj C\" asked how the parameters for IEEE Standard 754 floating point arithmetic were chosen.  I replied that I didn't know but would try to find out.  I called emeritus U. C. Berkeley Professor W. (Velvel) Kahan, who was the principle architect of 754.  Here is what I learned.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#ac0b6ec9-08a0-484c-8d98-6edb888c2a14\">The Zoo<\/a><\/li><li><a href=\"#e79ad6a5-d2a0-4b3d-95bb-ec8ca01f323e\">IEEE 754<\/a><\/li><li><a href=\"#43497456-a016-4e14-ad5a-e4c9db08b796\">Kahan's Parameters<\/a><\/li><li><a href=\"#69b44b4d-b2cb-4e06-8519-054132224ab0\">Retrospect<\/a><\/li><\/ul><\/div><h4>The Zoo<a name=\"ac0b6ec9-08a0-484c-8d98-6edb888c2a14\"><\/a><\/h4><p>First, some background. George Forsythe, Mike Malcolm and I wrote a <a href=\"https:\/\/www.amazon.com\/Mathematical-Computations-Prentice-Hall-automatic-computation\/dp\/0131653326\">textbook published in 1977<\/a> that began with a chapter on floating point computation.  The chapter includes the following table of the floating point parameters for various computers.  The only copy of the original book that I have nearby has some handwritten corrections, so I scanned the table in the Greek translation.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/fmm.png\" alt=\"\"> <\/p><p>This was before personal computers and IEEE 754.  It was a real zoo. There were binary and decimal and even a Russian ternary machine. There were a variety of word lengths.  Among the binary machines, there were base 2, base 8 and base 16 normalizations.<\/p><h4>IEEE 754<a name=\"e79ad6a5-d2a0-4b3d-95bb-ec8ca01f323e\"><\/a><\/h4><p>Around 1976 Intel began development of the 8086 microprocessor, which would become the CPU chip in the first IBM PCs.  John Palmer (who had been a student of mine in New Mexico) was in charge of the design of the <a href=\"https:\/\/www.computer.org\/csdl\/proceedings\/afips\/1980\/5088\/00\/50880887.pdf\">8087 floating point coprocessor<\/a>.  He recommended that Kahan become a consultant.  Palmer and Kahan convinced Intel to take the unprecedented step of making their design public so that it could be an industry wide standard.  That Intel specification became the basis for IEEE 754.<\/p><p>I wrote a two-part series in this blog about 754 <a href=\"https:\/\/blogs.mathworks.com\/cleve\/2014\/07\/07\/floating-point-numbers\/\">floating point<\/a> and <a href=\"https:\/\/blogs.mathworks.com\/cleve\/2014\/07\/21\/floating-point-denormals-insignificant-but-controversial-2\">denormals<\/a>.<\/p><h4>Kahan's Parameters<a name=\"43497456-a016-4e14-ad5a-e4c9db08b796\"><\/a><\/h4><p>Kahan told me that he chose the design parameters for the 8087 and consequently for the 754 standard.  He said he looked at industry practice at the time because he wanted existing software to move to the new processor with a minimum of difficulty.  The 32- and 64- bit word lengths were dictated by the x86 design.  There were two architectures that came close to his vision, the IBM 360\/370 and the DEC PDP-11\/VAX.<\/p><p>So, the zoo was reduced to these two.  Here are their floating point radix <tt>beta<\/tt> and exponent and fraction lengths.<\/p><pre>                      single         double\r\n             beta   exp  frac      exp  frac<\/pre><pre>   IBM 360    16      7    24       11    52<\/pre><pre>   DEC VAX F   2      8    23\r\n   DEC VAX D   2                     8    55\r\n   DEC VAX G   2                    11    52<\/pre><pre>   IEEE 754    2      8    23       11    52<\/pre><p>Nobody was happy with the base-16 normalization on the IBM.  Argonne's Jim Cody called it \"wobbling precision\".  The DEC VAX had two double precision formats.  The D format had the same narrow exponent range as single precision.  That was unfortunate.  As I remember it, DEC's G format with three more bits in the exponent was introduced later, possibly influenced by the ongoing discussions in the IEEE committee.<\/p><p>The VAX F format for single and G format for double have the same bit lengths as the Palmer\/Kahan proposal, but the exponent bias is different, so the resulting <tt>realmax<\/tt> and <tt>realmin<\/tt> differ by a factor of four.  DEC lobbied for some flexibility in that part of the standard, but eventually they had to settle for not quite conforming.<\/p><h4>Retrospect<a name=\"69b44b4d-b2cb-4e06-8519-054132224ab0\"><\/a><\/h4><p>Looking back after 40 years, the IEEE 754 was a remarkable achievement. Today, everybody gratefully accepts it.  The zoo has been eliminated. Only the recent introduction of a second format of half precision has us thinking about these questions again.<\/p><p>Thanks, Raj, for asking.<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_0f05f6530875415595b377624ca820fd() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='0f05f6530875415595b377624ca820fd ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 0f05f6530875415595b377624ca820fd';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2019 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_0f05f6530875415595b377624ca820fd()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2018b<br><\/p><\/div><!--\r\n0f05f6530875415595b377624ca820fd ##### SOURCE BEGIN #####\r\n%% Floating Point Arithmetic Before IEEE 754\r\n% In a comment following\r\n% <https:\/\/blogs.mathworks.com\/cleve\/2019\/01\/16\/variable-format-half-precision-floating-point-arithmetic\/\r\n% my post about half-precision arithmetic>, \"Raj C\" asked how the parameters\r\n% for IEEE Standard 754 floating point arithmetic were chosen.  I replied\r\n% that I didn't know but would try to find out.  I called emeritus\r\n% U. C. Berkeley Professor W. (Velvel) Kahan, who was the principle\r\n% architect of 754.  Here is what I learned.\r\n\r\n%% The Zoo\r\n% First, some background.\r\n% George Forsythe, Mike Malcolm and I wrote a\r\n% <https:\/\/www.amazon.com\/Mathematical-Computations-Prentice-Hall-automatic-computation\/dp\/0131653326\r\n% textbook published in 1977> that began with a chapter on floating\r\n% point computation.  The chapter includes the following table of the\r\n% floating point parameters for various computers.  The only copy of the\r\n% original book that I have nearby has some handwritten corrections, so I\r\n% scanned the table in the Greek translation.\r\n%\r\n% <<fmm.png>>\r\n%\r\n% This was before personal computers and IEEE 754.  It was a real zoo.\r\n% There were binary and decimal and even a Russian ternary machine. \r\n% There were a variety of word lengths.  Among the binary machines,\r\n% there were base 2, base 8 and base 16 normalizations.\r\n\r\n%% IEEE 754\r\n% Around 1976 Intel began development of the 8086 microprocessor, which\r\n% would become the CPU chip in the first IBM PCs.  John Palmer (who had\r\n% been a student of mine in New Mexico) was in charge of the design of the\r\n% <https:\/\/www.computer.org\/csdl\/proceedings\/afips\/1980\/5088\/00\/50880887.pdf\r\n% 8087 floating point coprocessor>.  He recommended that Kahan become a\r\n% consultant.  Palmer and Kahan convinced Intel to take the unprecedented\r\n% step of making their design public so that it could be an industry wide\r\n% standard.  That Intel specification became the basis for IEEE 754.\r\n%\r\n% I wrote a two-part series in this blog about 754\r\n% <https:\/\/blogs.mathworks.com\/cleve\/2014\/07\/07\/floating-point-numbers\/ \r\n% floating point> and\r\n% <https:\/\/blogs.mathworks.com\/cleve\/2014\/07\/21\/floating-point-denormals-insignificant-but-controversial-2\r\n% denormals>.\r\n\r\n%% Kahan's Parameters\r\n% Kahan told me that he chose the design parameters for the 8087 and\r\n% consequently for the 754 standard.  He said he looked at industry\r\n% practice at the time because he wanted existing software to move\r\n% to the new processor with a minimum of difficulty.  The 32- and 64-\r\n% bit word lengths were dictated by the x86 design.  There were\r\n% two architectures that came close to his vision, the IBM 360\/370\r\n% and the DEC PDP-11\/VAX.\r\n%\r\n% So, the zoo was reduced to these two.  Here are their floating point\r\n% radix |beta| and exponent and fraction lengths.\r\n%\r\n%                        single         double\r\n%               beta   exp  frac      exp  frac\r\n%\r\n%     IBM 360    16      7    24       11    52\r\n%\r\n%     DEC VAX F   2      8    23  \r\n%     DEC VAX D   2                     8    55\r\n%     DEC VAX G   2                    11    52\r\n%\r\n%     IEEE 754    2      8    23       11    52\r\n%\r\n% Nobody was happy with the base-16 normalization on the IBM.  Argonne's\r\n% Jim Cody called it \"wobbling precision\".  The DEC VAX had two\r\n% double precision formats.  The D format had the same narrow exponent \r\n% range as single precision.  That was unfortunate.  As I remember it,\r\n% DEC's G format with three more bits in the exponent was introduced later,\r\n% possibly influenced by the ongoing discussions in the IEEE committee.\r\n%\r\n% The VAX F format for single and G format for double have the same\r\n% bit lengths as the Palmer\/Kahan proposal, but the exponent bias\r\n% is different, so the resulting |realmax| and |realmin| differ by\r\n% a factor of four.  DEC lobbied for some flexibility in that part\r\n% of the standard, but eventually they had to settle for not quite\r\n% conforming.\r\n\r\n%% Retrospect\r\n% Looking back after 40 years, the IEEE 754 was a remarkable achievement.\r\n% Today, everybody gratefully accepts it.  The zoo has been eliminated.\r\n% Only the recent introduction of a second format of half precision\r\n% has us thinking about these questions again.\r\n%\r\n% Thanks, Raj, for asking.\r\n\r\n##### SOURCE END ##### 0f05f6530875415595b377624ca820fd\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"http:\/\/blogs.mathworks.com\/cleve\/files\/fmm.png\" onError=\"this.style.display ='none';\" \/><\/div><!--introduction--><p>In a comment following <a href=\"https:\/\/blogs.mathworks.com\/cleve\/2019\/01\/16\/variable-format-half-precision-floating-point-arithmetic\/\">my post about half-precision arithmetic<\/a>, \"Raj C\" asked how the parameters for IEEE Standard 754 floating point arithmetic were chosen.  I replied that I didn't know but would try to find out.  I called emeritus U. C. Berkeley Professor W. (Velvel) Kahan, who was the principle architect of 754.  Here is what I learned.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/cleve\/2019\/01\/18\/floating-point-arithmetic-before-ieee-754\/\">read more >><\/a><\/p>","protected":false},"author":78,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16,8,7],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/4410"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/comments?post=4410"}],"version-history":[{"count":1,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/4410\/revisions"}],"predecessor-version":[{"id":4412,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/4410\/revisions\/4412"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/media?parent=4410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/categories?post=4410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/tags?post=4410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}