Matrices

fresh off the press from my collaborator Andrzej Ehrenfeucht

I think that the formula for approximately solving n linear equations with k variables, where n is greater than or equal to k, is the most important formula in the theory of matrices.

[A] an n by k matrix of coefficients;

[X] a k by 1 matrix (column) of variables;

[B] an n by 1 matrix of right-hand sides of equations;

[A]^{T} a k by n
transposed matrix [A].

(In a transposed matrix, rows become columns and columns become rows.)

[X] = ([A]^{T} * [A])^{-1} * [A]^{T} *
[B]

When n = k, [A] and [A]^{T}
are square matrices, so we have,

([A]^{T} * [A])^{-1} = [A]^{-1} * [A]^{T}^{-1}

So,

[X] = ([A]^{T} * [A])^{-1}
* [A]^{T} * [B]

[A]^{-1} *
[A]^{T-1} * [A]^{T} * [B]

[A]^{-1} *
[B] because, ([A]^{T})^{-1}
* [A]^{T} = I,

which is a formula for solving n linear equations with n variables.

The "best approximation" is defined as the minimal value of

Ss = sum(([A][X] - [B])^{2})

(the minimum of the sum of squares)

The formula is derived from this definition. Let the variables be x_{1}, ..., x_{k}. When the Ss have a minimum, then

dSs/dx_{j} = 0 for j = 1,..., k

This is a system of k equations with k variables. When it is written explicitly in terms of [A], [X], and [B] (which requires quite messy algebra which is difficult for many students), we get

([A]^{T} * [A]) * [X] = [A]^{T}
* [B]

which is solved by multiplying both
sides by ([A]^{T} * [A])^{-1}.

[X] = ([A]^{T} * [A])^{-1} * [A]^{T} *
[B]

A specific application of this formula is when the x's are coefficients of a polynomial of degree k-1. The best fitting polynomial of first degree describes linear regression, the best fitting quadratic polynomial describes quadratic regression, and so on.

Remark.

I do not know why this formula is not taught in all courses of linear algebra. Maybe it is so because these courses are taught "geometrically" and therefore are restricted to at most 3 by 3 matrices. Or because linear algebra is taught before calculus, so the least square method, which requires calculus, is banned.

Calculus Index