Linear algebra
Textbooks
Matrix properties
Span
If vector is in the span of a set of vectors , it means that it can be recreated by a linear combination of those vectors.
Singular matrix
- A matrix which removes some of the dimensions as part of its transformation.
- Has a null space (different inputs produce the same output).
Positive definite matrix
- This corresponds to a "bowl-shaped" matrix.
- Real PDMs do not have to be symmetric.
- Complex PDMs have to be Hermitian (or self-adjoint).
- All eigenvalues are positive (as long as it is a complex Hermitian matrix or a real symmetric matrix).
- Non-singular (invertible).
Positive semidefinite matrix
- This corresponds to a matrix with a ridge of infinite length.
- Real PDMs do not have to be symmetric.
- Complex PDMs have to be Hermitian (or self-adjoint).
- All eigenvalues are non-negative (as long as it is a complex Hermitian matrix or a real symmetric matrix).
Real symmetric matrix
- Always has real eigenvalues.
- Possible to choose a complete set of perpendicular eigenvectors.
- Not necessarily invertible.
Hermitian matrix
- Equal to it's own conjugate transform.
- Complex positive (semi)definite matrices are Hermitian.
- Always has real eigenvalues (just like real symmetric matrices).
Gramm matrix
A matrix where every element is a dot product in some feature space.
Hilbert space
- Euclidean space generalized to infinite dimensions
- Can contain functions, since a function is pretty much the same as a vector of infinite length.
- Complete, meaning there are no "holes". For example, rational numbers are not complete because is not in it. Complete means you can work with limits of sequences.
- Vectors in Hilbert space have a concept of length and inner product.
- Consider the set of all sine waves which fit an integer amount of times on a string. You can create an arbitrary function by combining these together with appropriate coefficients. So we can consider each sine wave to each be an independent dimension in a Hilbert space. The function is now a point in this abstract space.
Reproducing kernel Hilbert space
A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space H with a reproducing kernel whose span is dense in . We could equivalently define an RKHS as a Hilbert space of functions with all evaluation functionals bounded and linear.
- Introduction to Machine Learning, short course on kernel methods
- Reproducing kernel Hilbert spaces in Machine Learning
Matrix transformations
Conjugate transform
Take a transform and a complex-conjugate of imaginary numbers.
Direct sum
Tensor product
- Mapping which takes two inputs and produces a real number.
- Tensor is a map which takes elements from the Cartesian space of vectors and produces a real number.
- Tensorts come in many forms because there are many different ways of making Cartesian products?
- http://hitoshi.berkeley.edu/221a/tensorproduct.pdf.
Eigendecomposition
- Represent a matrix in terms of its eigenvalues and eigenvectors.
- Real symmetric matrices have real eigenvalues and orthogonal eigenvectors.
- Singular value decomposition extends this to non-square matrices.
- Wikipedia