CR and CAB, Rank Revealing Matrix Factorizations
The rank of a linear transformation is a fundamental concept in linear algebra and matrix factorizations are fundamental concepts in numerical linear algebra. Gil Strang's 2020 Vision of Linear Algebra seeks to introduce these notions early in an introductory linear algebra course.
The CR matrix factorization provides a view of rref, the reduced row echelon form, as a rank revealing matrix factorization. I discussed CR in a pair of posts in October. I now want to describe the CAB factorization, which uses rref twice in order to treat both rows and columns in the same way.
Gil and I are writing a review paper about these ideas, LU and CR Elimination. Here is a link to the current draft, LUCR_43.pdf.
Contents
CAB
This is a quick description of cab.
[C,W,B] = cab(A) returns C = A(:,cols), B = A(rows,:), and W = A(rows,cols) so that
A = C*inv(W)*B
[C,W,B,cols,rows] = cab(A) also returns the indices cols and rows.
All the elements of C, W and B are taken from the input matrix A itself; C is some of its columns, B is some of its rows, and W is the set intersection of C and B. Actually, cols and rows are all this factorization computes. The column indices come from the reduced row echelon form, rref(A), and the row indices from the transpose, rref(A').
Rank
The fact that the number of linearly independent columns of a matrix is equal to the number of linearly independent rows is not trivial and requires proof. This number is the rank of the matrix and is the size of the nonsingular W.
W is r-by-r where rank(A) = r = length(cols) = length(rows).
If you try to find C, W and R with a W that is too small, then C*inv(W)*R will not be able to reproduce A. If W is too large, it will be singular.
Rank one
If A has rank one, A = u*v' for column vector u and row vector v'. Then C is a multiple of u, B is a multiple of v', and W is the first nonzero element in u or v. Here is an example where the first row is deliberately zero.
A = (0:3)'*(4:6)
A =
0 0 0
4 5 6
8 10 12
12 15 18
format compact
[C,W,B] = cab(A)
C =
0
4
8
12
W =
4
B =
4 5 6
Rank two
In this example, even casual inspection will reveal the rank. Note that A is rectangular.
A = reshape(1:24,4,6)'
A =
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
[C,W,B,cols,rows] = cab(A)
C =
1 2
5 6
9 10
13 14
17 18
21 22
W =
1 2
5 6
B =
1 2 3 4
5 6 7 8
cols =
1 2
rows =
1 2
rank = length(cols)
rank =
2
Full rank
If A is square and nonsingular, then C = B = W = A.
For example,
A = pascal(3)
A =
1 1 1
1 2 3
1 3 6
[C,W,B] = cab(A)
C =
1 1 1
1 2 3
1 3 6
W =
1 1 1
1 2 3
1 3 6
B =
1 1 1
1 2 3
1 3 6
Franklin magic square
The Franklin semimagic square is attributed to Benjamin Franklin in about 1752. A letter from Franklin to one of his colleagues is included in the Franklin papers referenced below.
All the rows and columns of Franklin's square have the required magic sum, but the diagonals do not, so the matrix isn't fully magic. However, many other interesting submatrices are magic, including bent diagonals and any eight elements arranged symmetrically about the center.
What is the rank of this 8-by-8 matrix?
A = franklin(8)
A =
52 61 4 13 20 29 36 45
14 3 62 51 46 35 30 19
53 60 5 12 21 28 37 44
11 6 59 54 43 38 27 22
55 58 7 10 23 26 39 42
9 8 57 56 41 40 25 24
50 63 2 15 18 31 34 47
16 1 64 49 48 33 32 17
It is hard to think of Franklin's square as a linear transformation, but cab can still compute its rank.
[C,W,B] = cab(A)
C =
52 61 4
14 3 62
53 60 5
11 6 59
55 58 7
9 8 57
50 63 2
16 1 64
W =
52 61 4
14 3 62
53 60 5
B =
52 61 4 13 20 29 36 45
14 3 62 51 46 35 30 19
53 60 5 12 21 28 37 44
So, the rank is only three.
Computing C*inv(W)*B introduces some roundoff error.
test = C*inv(W)*B;
displayer('test')
test =
52.00 61.00 4.00 13.00 20.00 29.00 36.00 45.00
14.00 3.00 62.00 51.00 46.00 35.00 30.00 19.00
53.00 60.00 5.00 12.00 21.00 28.00 37.00 44.00
11.00 6.00 59.00 54.00 43.00 38.00 27.00 22.00
55.00 58.00 7.00 10.00 23.00 26.00 39.00 42.00
9.00 8.00 57.00 56.00 41.00 40.00 25.00 24.00
50.00 63.00 2.00 15.00 18.00 31.00 34.00 47.00
16.00 1.00 64.00 49.00 48.00 33.00 32.00 17.00
Rounding test to the nearest integers reproduces Franklin's magic square exactly.
Backslash and forward slash
Computing C*inv(W)*B by (C/W)*B or C*(W\B) is usually more accurate. If A has integer elements, you might try round(C/W*B) or round(C*(W\B)).
This 5-by-5 matrix was deliberately constructed so that its rank deficiency is not obvious.
A = gallery(5)
A =
-9 11 -21 63 -252
70 -69 141 -421 1684
-575 575 -1149 3451 -13801
3891 -3891 7782 -23345 93365
1024 -1024 2048 -6144 24572
[C,W,B] = cab(A)
C =
-9 11 -21 63
70 -69 141 -421
-575 575 -1149 3451
3891 -3891 7782 -23345
1024 -1024 2048 -6144
W =
-9 11 -21 63
70 -69 141 -421
-575 575 -1149 3451
3891 -3891 7782 -23345
B =
-9 11 -21 63 -252
70 -69 141 -421 1684
-575 575 -1149 3451 -13801
3891 -3891 7782 -23345 93365
rank = size(W,1)
rank =
4
In this example, W is mildly ill-conditioned.
kappa = cond(W)
kappa = 2.4301e+04
There are three ways to recreate A.
invw = C*inv(W)*B; back = (C/W) * B; fore = C * (W\B);
The relative error with inversion is affected by the condition of W, while backslash and forward slash are almost five orders of magnitude more accurate,
format short e relerr = [norm(A-invw) norm(A-fore) norm(A-back)]/norm(A)
relerr = 2.1266e-13 4.7760e-18 3.7217e-17
Rounding any of the three to integers reproduces A exactly.
A = round(C*inv(W)*B)
A =
-9 11 -21 63 -252
70 -69 141 -421 1684
-575 575 -1149 3451 -13801
3891 -3891 7782 -23345 93365
1024 -1024 2048 -6144 24572
Gauss-Jordan
The algorithm for computing the rref is Gauss-Jordan elimination, which has a dubious reputation in the numerical analysis community. After listening to me trying to characterize the algorithm's notoriety, Gil wrote in the introductory paragraphs of our survey that Gauss-Jordan is "expensive and numerically perilous".
The "expensive" part is easy to explain. The computation of the rref introduces zeros in a column both above and below the pivot element, while Gaussian elimination only zeros below the pivot. That's less work. We can measure computational complexity in terms of a fused multiply-add instruction, FMA, that multiplies two floating point values and then adds a third value to the result in one operation. For a large n-by-n system, Gauss-Jordan requires about (1/2) n^3 FMAs, while Gaussian elimination requires less, only (1/3) n^3 FMAs.
However, we don't intend CR and CAB to be used with large matrices. For our purposes, even using Gauss-Jordan twice is not expensive.
The "numerically perilous" portion of Gauss-Jordan's reputation is more subtle. It derives from solving a linear system A*x = b by computing the rref of an augmented matrix, [A b]. Gaussian elimination, on the other hand, finds triangular factors L and U and then computes x by back substitution. We could say that Gaussian elimination is partial elimination, while Gauss-Jordan is complete elimination. The two approaches are compared in the 1975 paper by Jim Wilkinson and Gwen Peters referenced below. Gaussian elimination almost always computes the solution of a nearby system, but the same cannot be said for Gauss-Jordan on the augmented matrix.
However, we are not using either form of elimination to actually compute the CAB factors; rref just selects indices. The matrices C, W and B from CAB(A) are submatrices of A. They are certainly accurate. There are two potential numerical difficulties -- W might be badly conditioned or it might be the wrong size.
Condition
What if A is badly conditioned? Then W has to inherit that conditioning. Here is an example. Start with
n = 12; U = eye(n,n) - triu(ones(n,n),1)
U =
1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
0 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
0 0 1 -1 -1 -1 -1 -1 -1 -1 -1 -1
0 0 0 1 -1 -1 -1 -1 -1 -1 -1 -1
0 0 0 0 1 -1 -1 -1 -1 -1 -1 -1
0 0 0 0 0 1 -1 -1 -1 -1 -1 -1
0 0 0 0 0 0 1 -1 -1 -1 -1 -1
0 0 0 0 0 0 0 1 -1 -1 -1 -1
0 0 0 0 0 0 0 0 1 -1 -1 -1
0 0 0 0 0 0 0 0 0 1 -1 -1
0 0 0 0 0 0 0 0 0 0 1 -1
0 0 0 0 0 0 0 0 0 0 0 1
U is badly conditioned. cond(U,1) and cond(U,inf) are both equal to n*2^(n-1).
Let
A = U'*U
A =
1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
-1 2 0 0 0 0 0 0 0 0 0 0
-1 0 3 1 1 1 1 1 1 1 1 1
-1 0 1 4 2 2 2 2 2 2 2 2
-1 0 1 2 5 3 3 3 3 3 3 3
-1 0 1 2 3 6 4 4 4 4 4 4
-1 0 1 2 3 4 7 5 5 5 5 5
-1 0 1 2 3 4 5 8 6 6 6 6
-1 0 1 2 3 4 5 6 9 7 7 7
-1 0 1 2 3 4 5 6 7 10 8 8
-1 0 1 2 3 4 5 6 7 8 11 9
-1 0 1 2 3 4 5 6 7 8 9 12
A is symmetric, positive definite and full rank. The determinant of A is equal to one. I don't see a neat formula for cond(A), but I know it does grow like 4^n.
Since A is full rank, its A = C*inv(W)*B factorization must be A = A*inv(A)*A.
[C,W,B] = cab(A); test = [isequal(C,A) isequal(W,A) isequal(B,A)]
test = 1×3 logical array 1 1 1
W is badly conditioned. inv(W) has large elements.
Winv = inv(W)
Winv =
Columns 1 through 6
1398102 699051 349526 174764 87384 43696
699051 349526 174763 87382 43692 21848
349526 174763 87382 43691 21846 10924
174764 87382 43691 21846 10923 5462
87384 43692 21846 10923 5462 2731
43696 21848 10924 5462 2731 1366
21856 10928 5464 2732 1366 683
10944 5472 2736 1368 684 342
5504 2752 1376 688 344 172
2816 1408 704 352 176 88
1536 768 384 192 96 48
1024 512 256 128 64 32
Columns 7 through 12
21856 10944 5504 2816 1536 1024
10928 5472 2752 1408 768 512
5464 2736 1376 704 384 256
2732 1368 688 352 192 128
1366 684 344 176 96 64
683 342 172 88 48 32
342 171 86 44 24 16
171 86 43 22 12 8
86 43 22 11 6 4
44 22 11 6 3 2
24 12 6 3 2 1
16 8 4 2 1 1
logWinv = floor(log2(Winv))
logWinv =
20 19 18 17 16 15 14 13 12 11 10 10
19 18 17 16 15 14 13 12 11 10 9 9
18 17 16 15 14 13 12 11 10 9 8 8
17 16 15 14 13 12 11 10 9 8 7 7
16 15 14 13 12 11 10 9 8 7 6 6
15 14 13 12 11 10 9 8 7 6 5 5
14 13 12 11 10 9 8 7 6 5 4 4
13 12 11 10 9 8 7 6 5 4 3 3
12 11 10 9 8 7 6 5 4 3 2 2
11 10 9 8 7 6 5 4 3 2 1 1
10 9 8 7 6 5 4 3 2 1 1 0
10 9 8 7 6 5 4 3 2 1 0 0
Follow the diagonal of logWinv from (n,n) to (1,1).
Winv(1,1) > 4^(n-2)
ans = logical 1
Is A close to a matrix of lower rank? The answer is yes, but the lower rank approximation comes from the SVD.
CAB vs. SVD
The SVD is the ultimate rank revealing factorization, but it does not actually attempt to compute the rank.
[U,S,V] = svd(A)
computes a full set of singular values and vectors as if the matrix had full rank. The singular values, s = diag(S), are indexed in decreasing order,
s(1) >= . . . >= s(r) >= s(r+1) . . .
You can compare the singular values with a tolerance to obtain an approximate rank and factorization.
The choice of an effective rank is backed by the Eckart-Young-Mirsky theorem: If Ar is the approximation made by choosing the rank to be r,
Ar = U(:,1:r)*S(1:r,1:r)*V(:,1:r)'
the norm of the error is bounded by the first term that is omitted,
norm(A - Ar,2) <= s(r+1)
If the singular values decrease rapidly, an accurate low rank approximation is achievable with only a few terms.
On the other hand, CAB chooses bases from the columns and rows of a matrix itself. There is no Eckert-Young theorem for the CR or CAB factorizations.
We intend CR and CAB to be used in the classroom on small matrices, usually with integer entries. We will let the SVD and randomized matrix factorizations handle the big stuff.
References
Benjamin Franklin, The Papers of Benjamin Franklin, Volume 44, 1750-53, To Peter Collinson. (American Philosophical Society and Yale University) p. 392. https://franklinpapers.org/framedVolumes.jsp?vol=4&page=392a.
G. Peters and J. H. Wilkinson, "On the stability of Gauss-Jordan elimination with pivoting", Communications ACM 18, 1975, p. 20-24. https://dl.acm.org/doi/10.1145/360569.360653, also at http://www.stat.uchicago.edu/~lekheng/courses/302/gauss-jordan2.pdf
Postscript
Ben Franklin can also supply the graphic for today's post.
A = franklin(8);
[~,~,B] = cab(A);
plot(B)
Comments
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.