Hilbert Matrices 6

Posted by Cleve Moler,

I first encountered the Hilbert matrix when I was doing individual studies under Professor John Todd at Caltech in 1960. It has been part of my professional life ever since.

Contents

David Hilbert

Around the turn of the 20th century, David Hilbert was the world's most famous mathematician. He introduced the matrix that now bears his name in a paper in 1895. The elements of the matrix, which are reciprocals of consecutive positive integers, are constant along the antidiagonals.

$$ h_{i,j} = \frac{1}{i+j-1}, \ \ i,j = 1:n $$

    format rat
    H5 = hilb(5)
H5 =
       1              1/2            1/3            1/4            1/5     
       1/2            1/3            1/4            1/5            1/6     
       1/3            1/4            1/5            1/6            1/7     
       1/4            1/5            1/6            1/7            1/8     
       1/5            1/6            1/7            1/8            1/9     
    latex(sym(H5));

$$ H_5 = \left(\begin{array}{ccccc} 1 & \frac{1}{2} & \frac{1}{3} & \frac{1}{4} & \frac{1}{5}\\ \frac{1}{2} & \frac{1}{3} & \frac{1}{4} & \frac{1}{5} & \frac{1}{6}\\ \frac{1}{3} & \frac{1}{4} & \frac{1}{5} & \frac{1}{6} & \frac{1}{7}\\ \frac{1}{4} & \frac{1}{5} & \frac{1}{6} & \frac{1}{7} & \frac{1}{8}\\ \frac{1}{5} & \frac{1}{6} & \frac{1}{7} & \frac{1}{8} & \frac{1}{9} \end{array}\right) $$

Here's a picture. The continuous surface generated is very smooth.

    H12 = hilb(12);
    surf(log(H12))
    view(60,60)

Least squares

Hilbert was interested in this matrix because it comes up in the least squares approximation of a continuous function on the unit interval by polynomials, expressed in the conventional basis as linear combinations of monomials.

$$ p(x) = \sum_{j=1}^n c_j x^{j-1} $$

The coefficient matrix for the normal equations has elements

$$ \int_0^1 x^{i+j-2} dx \ = \ \frac{1}{i+j-1} $$

Properties

A Hilbert matrix has many useful properties.

  • Symmetric.
  • Positive definite.
  • Hankel, $a_{i,j}$ is a function of $i+j$.
  • Cauchy, $a_{i,j} = 1/(x_i - y_j)$.
  • Nearly singular.

Each column of a Hilbert matrix is nearly a multiple of the other columns. So the columns are nearly linearly dependent and the matrix is close to, but not exactly, singular.

hilb

MATLAB has always had functions hilb and invhilb that compute the Hilbert matrix and its inverse. The body of hilb is now only two lines.

J = 1:n;
H = 1./(J'+J-1);

We often cite this as a good example of singleton expansion. A column vector is added to a row vector to produce a matrix, then a scalar is subtracted from that matrix, and finally the reciprocals of the elements produce the result.

Inverse

It is possible to express the elements of the inverse of the Hilbert matrix in terms of binomial coefficients. For reasons that I've now forgotten, I always use $T$ for $H^{-1}$.

$$ t_{i,j} = (-1)^{i+j} (i+j-1) {{n+i-1} \choose {n-j}} {{n+j-1} \choose {n-i}} {{i+j-2} \choose {i-1}}^2 $$

The elements of the inverse Hilbert matrix are integers, but they are large integers. The largest ones are on the diagonal. For order 13, the largest element is

$$ (T_{13})_{9,9} \ = \ 100863567447142500 $$

This is over $10^{17}$ and is larger than double precision flintmax.

    format longe
    flintmax = flintmax
flintmax =
     9.007199254740992e+15

So, it is possible to represent the largest elements exactly only if $n \le 12$.

The HELP entry for invhilb includes a sentence inherited from my original Fortran MATLAB.

  The result is exact for  N  less than about 15.

Now that's misleading. It should say

  The result is exact for  N  <= 12.

(I'm filing a bug report.)

invhilv

The body of invhilb begins by setting p to the order n. The doubly nested for loops then evaluate the binomial coefficient formula recursively, avoiding unnecessary integer overflow.

    dbtype invhilb 18:28
18    p = n;
19    for i = 1:n
20        r = p*p;
21        H(i,i) = r/(2*i-1);
22        for j = i+1:n
23            r = -((n-j+1)*r*(n+j-1))/(j-1)^2;
24            H(i,j) = r/(i+j-1);
25            H(j,i) = r/(i+j-1);
26        end
27        p = ((n-i)*p*(n+i))/(i^2);
28    end

I first programmed this algorithm in machine language for the Burroughs 205 Datatron at Caltech almost 60 years ago. I was barely out of my teens.

Here's the result for n = 6.

    format short
    T6 = invhilb(6)
T6 =
          36        -630        3360       -7560        7560       -2772
        -630       14700      -88200      211680     -220500       83160
        3360      -88200      564480    -1411200     1512000     -582120
       -7560      211680    -1411200     3628800    -3969000     1552320
        7560     -220500     1512000    -3969000     4410000    -1746360
       -2772       83160     -582120     1552320    -1746360      698544

A checkerboard sign pattern with large integers in the inverse cancels the smooth surface of the Hilbert matrix itself.

    T12 = invhilb(12);
    spy(T12 > 0)

A log scale mitigates the jaggedness.

    surf(sign(T12).*log(abs(T12)))
    view(60,60)

Rookie experiment

Now using MATLAB, I am going to repeat the experiment that I did on the Burroughs 205 when I was still a rookie. I had just written my first program that used Gaussian elimination to invert matrices. I proceeded to test it by inverting Hilbert matrices and comparing the results with the exact inverses. (Today's code uses this utility function that picks out the largest element in a matrix.)

    maxabs = @(X) max(double(abs(X(:))));

Here is n = 10.

    n = 10
    H = hilb(n);
    X = inv(H);        % Computed inverse
    T = invhilb(n);    % Theoretical inverse
    E = X - T;         % The error
    err = maxabs(E)
n =
    10
err =
   5.0259e+08

At first I might have said:

Wow! The error is $10^8$. That's a pretty big error. Can I trust my matrix inversion code? What went wrong?

But I knew the elements of the inverse are huge. We should be looking at relative error.

   relerr = maxabs(E)/maxabs(T)
relerr =
   1.4439e-04

OK. The relative error is $10^{-4}$. That still seems like a lot.

I knew that the Hilbert matrix is nearly singular. That's why I was using it. John Todd was one of the first people to write about condition numbers. An error estimate that involves nearness to singularity and the floating point accuracy would be expressed today by

   esterr = cond(H)*eps
esterr =
    0.0036

That was about all I understood at the time. The roundoff error in the inversion process is magnified by the condition number of the matrix. And, the error I observe is less than the estimate that this simple analysis provides. So my inversion code passes this test.

Deeper explanation

I met Jim Wilkinson a few years later and came to realize that there is more to the story. I'm not actually inverting the Hilbert matrix. There are roundoff errors involved in computing H even before it is passed to the inversion routine.

Today, the Symbolic Math Toolbox helps provide a deeper explanation. The 'f' flag on the sym constructor says to convert double precision floating point arguments exactly to their rational representation. Here's how it works in this situation. To save space, I'll print just the first column.

    H = hilb(n);
    F = sym(H,'f');
    F(:,1)
ans =
                                  1
                                1/2
 6004799503160661/18014398509481984
                                1/4
 3602879701896397/18014398509481984
 6004799503160661/36028797018963968
 2573485501354569/18014398509481984
                                1/8
 2001599834386887/18014398509481984
 3602879701896397/36028797018963968

The elements of H that are not exact binary fractions become ratios of large integers. The denominators are powers of two; in this case $2^{54}$ and $2^{55}$. The numerators are these denominators divided by $3$, $5$, etc. and then rounded to the nearest integer. The elements of F are as close to the exact elements of a Hilbert matrix as we can get in binary floating point.

Let's invert F, using the exact rational arithmetic provided by the Symbolic Toolbox. (I couldn't do this in 1960.)

    S = inv(F);

We now have three inverse Hilbert matrices, X, S, and T.

  • X is the approximate inverse computed with floating point arithmetic by the routine I was testing years ago, or by MATLAB inv function today.
  • S is the exact inverse of the floating point matrix that was actually passed to the inversion routine.
  • T is the exact Hilbert inverse, obtained from the binomial coefficient formula.

Let's print the first columns alongside each other.

   fprintf('%12s %16s %15s\n','X','S','T')
   fprintf('%16.4f %16.4f %12.0f\n',[X(:,1) S(:,1) T(:,1)]')
           X                S               T
         99.9961          99.9976          100
      -4949.6667       -4949.7926        -4950
      79192.8929       79195.5727        79200
    -600535.3362     -600559.6914      -600600
    2522211.3665     2522327.5182      2522520
   -6305451.1288    -6305770.4041     -6306300
    9608206.6797     9608730.4926      9609600
   -8750253.0592    -8750759.2546     -8751600
    4375092.6697     4375358.4162      4375800
    -923624.4113     -923682.8529      -923780

It looks like X is closer to S than S is to T. Let's confirm by computing two relative errors, the difference between X and S, and the difference between S and T.

    format shorte
    relerr(1) = maxabs(X - S)/maxabs(T);
    relerr(2) = maxabs(S - T)/maxabs(T)
    relerrtotal = sum(relerr)
relerr =
   5.4143e-05   9.0252e-05
relerrtotal =
   1.4439e-04

The error in the computed inverse comes from two sources -- generating the matrix in the first place and then computing the inverse. The first of these is actually larger than the second, although the two are comparable.


Get the MATLAB code

Published with MATLAB® R2017a

6 CommentsOldest to Newest

José-Javier Martínez replied on : 1 of 6

Dear Professor Moler: I would like to recall an additional useful property of the Hilbert matrix, namely its total positivity. Interestingly, this property is also related to the work of John Todd and Olga Taussky, since the reference given by Nick Higham in his “Test Matrix Toolbox for MATLAB” for the total positivity of Cauchy and Hilbert matrices was

-Olga Taussky and Marvin Marcus: Eigenvalues of finite matrices. In John Todd, editor, Survey of Numerical Analysis, pages 279—313. McGraw-Hill, New York, 1962.

The power of this property int the context of computing with high relative accuracy has been shown in

– Plamen Koev: Accurate computations with totally nonnegative matrices. SIAM J. Matrix Anal. Appl. 29(3), pp. 731-751 (2007).

Hi —

Thanks for the comment and the references. Koev’s paper is available from his web site:
https://math.mit.edu/~plamen/files/acctp.pdf

But maybe I didn’t make this point strongly enough — we can’t use these high accuracy algorithms, for two reasons. First, we only start with a floating point approximation to what would be a totally positive matrix. The roundoff errors made in generating the matrix in the first place have a bigger effect on the inverse than those generated during the inversion process. Second, to take advantage of totally positivity, it is necessary to have the representation as a product of exact, rational, bidiagonal matrices.

— Cleve

Michele replied on : 3 of 6

You made the remark that on the Burroughs 205, you did better on inverting the Hilber matrix than the condition number seemed to imply. cond(H) gives an upper estimate of the error, but this upper estimate can be way off. Suppose I have two matrices: the identity matrix and the diagonal matrix with elements 1,2,4,8,16…2^n; their condition number is 1 and 2^n respectively. I don’t see any reason why in the second case I would lose log10(2^n) digits.
Can you comment on when the condition number gives a tight estimate and whether there is a better estimator?

michele

Good question, Michele. The answer deserves a proper explanation, so I’ll make it the subject of a blog post. I’ve already got something cooking for next week, so it will be a couple of weeks before I get to this topic. Thanks for asking. — Cleve

Add A Comment

What is 1 + 7?

Preview: hide