Consider the following matrix which can be SVD decomposed as $$ \boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{bmatrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. $$
This is a \( 3\times 2 \) matrix which is decomposed in terms of a \( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?).
And the SVD decomposition (singular values) gives eigenvalues \( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the eigenvalues (singular values) are zero.
In the general case, where our design matrix \( \boldsymbol{X} \) has dimension \( n\times p \), the matrix is thus decomposed into an \( n\times n \) orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling the rest of the matrix. There are at most \( p \) singular values assuming that \( n > p \). In our regression examples for the nuclear masses and the equation of state this is indeed the case, while for the Ising model we have \( p > n \). These are often cases that lead to near singular or singular matrices.
The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.