A list of useful properties and their proofs in linear algebra. Some of them are also very useful in machine learning. Complex proofs are not in the scope of the post, but the references will be given if interested.
Let A and B be two square matrix, then
The determinant of a orthogonal matrix must be
Let be an matrix and let be its eigenvalues, then
Matrix is symmetric if
If is the eigen-value of , so is the conjugate of , denoted as
If is the eigen-vector of , so is the conjugate of , denoted as
Since is a real matrix,
has only real eigenvalues
Consider and are the eigne-value and eigen-vector of , repectively. Based on Proof 1.1,
Since is a vector and ,
is diagonalizable by an orthogonal matrix.
Every square matrix factors into where is upper triangular and . If has real eigenvalues then and can be chosen real: (a.k.a is an orthogonal matrix)
Based on Proof 1.2, all the eigen values of are real.
Based on Schur decomposition, .
Denote the diagonal matrix as , we have
If is nonsingular, is symmetric
Since is invertible,
Taking the transpose, we have
A real symmetric matrix is called positive definite if for all non-zero vectors .
The eigenvalues of a real symmetric positive-definite matrix are all positive.
Let be a (real) eigenvalue of and let be a corresponding real eigenvector. That is, we have
Then we multiply by on left and obtain,
The left hand side is positive as is positive definite and is a nonzero vector as it is an eigenvector.
Since the norm is positive, we must have is positive.
It follows that every eigenvalue of is real.
If eigenvalues of a real symmetric matrix are all positive, then is positive-definite.
Since Proof 1.3, where , we have
Putting , we can rewrite the above equation as
Then we have
By assumption eigenvalues are positive.
Also, since is a nonzero vector and is invertible, is not a zero vector.
Thus the sum expression above is positive, hence is positive for any nonzero vector .
Therefore, the matrix is positive-definite.
A is invertible
Since Proof 2.1, the matrix does not have 0 as an eigenvalue
We can prove this by contradiction:
If for some then by definition of eigenvalues (non-invertible), is an eigenvector with eigenvalue
We can prove this by using determinent
the inverse of A is positive-definite
All eigenvalues of are of the form , where is an eigenvalue of .
Since A is positive-definite, each eigenvalue is positive, hence is positive.
So all eigenvalues of are positive, and it yields that is positive-definite.
is scalar and is a column vector
and are column vectors
is matrix, is column vector and is scalar
If is symmetric, then
and are column vectors, is a matrix and is scalar
is a matrix
is a matrix and depends on
If is , then