updating regression slides

doc/src/Projects/2019/Exercises/addition.do.tx

@@ -1,25 +1,42 @@
-I think a possible way to show why $$\left \langle \hat u_i \right \rangle = 0$$ given that the columns of $$\hat X$$ is centered is by considering $$\left \langle \hat X \hat v_i \right \rangle$$:
+A possible way to show why $\left \langle \hat u_i \right \rangle = 0$
+given that the columns of $\hat X$ is centered is by considering
+$\left \langle \hat X \hat v_i \right \rangle$:
+
+!bt 
+
+\begin{align*} \left \langle \hat X \hat v_i \right \rangle &= \frac{1}{N}\sum_j ( \hat X \hat v_i )_j \\ &= \frac{1}{N}\sum_j \sum_k x_{jk}\hat v_i(k)\\ &= \frac{1}{N}\sum_k \hat v_i(k) \sum_j x_{jk} \\ &= \sum_k \hat v_i(k)\left( \frac{1}{N}\sum_j x_{jk} \right) \\ &= \sum_k \hat v_i(k) \left \langle \hat x_k \right \rangle
+\end{align*}
+!et
+
+where $x_{jk}$ being the element of $\hat X$ at row $j$ and column
+$k$, $( \hat X \hat v_i )_j $ the $j$-th element of the vector $\hat X
+\hat v_i $, $\hat x_k$ being the $k$-th column vector of $\hat X$, and
+$\hat v_i(k)$ the $k$-th element of the vector $\hat v_i$.
 
 
 
-$$\begin{align*} <br/>\left \langle \hat X \hat v_i \right \rangle &= \frac{1}{N}\sum_j ( \hat X \hat v_i )_j \\ <br/>&= \frac{1}{N}\sum_j \sum_k x_{jk}\hat v_i(k)\\ <br/>&= \frac{1}{N}\sum_k \hat v_i(k) \sum_j x_{jk} \\<br/>&= \sum_k \hat v_i(k)\left( \frac{1}{N}\sum_j x_{jk} \right) \\ <br/>&= \sum_k \hat v_i(k) \left \langle \hat x_k \right \rangle <br/>\end{align*}$$
-
-where $$x_{jk}$$ being the element of $$\hat X$$ at row $$j$$ and column $$k$$, $$( \hat X \hat v_i )_j $$ the $$j$$-th element of the vector $$\hat X \hat v_i $$, $$\hat x_k$$ being the $$k$$-th column vector of $$\hat X$$, and $$\hat v_i(k)$$ the $$k$$-th element of the vector $$\hat v_i$$.
+Since the columns of $\hat X$ are assumed to be centered, $\left
+\langle \hat x_k \right \rangle = 0$ for all $k$. This gives that
+$\left \langle \hat X \hat v_i \right \rangle = 0$.
 
 
 
-Since the columns of $$\hat X$$ are assumed to be centered, $$\left \langle \hat x_k \right \rangle = 0$$ for all $$k$$. This gives that $$\left \langle \hat X \hat v_i \right \rangle = 0$$.
+But $\left \langle \hat X \hat v_i \right \rangle = \left \langle \hat
+u_i d_i \right \rangle = d_i \left \langle \hat u_i \right \rangle $.
+
+Since $ \left \langle \hat X \hat v_i \right \rangle = 0$, then $d_i
+\left \langle \hat u_i \right \rangle = 0$ also. Assuming that $d_i
+\neq 0$ (otherwise the variance in the exercise would just be zero),
+gives that $\left \langle \hat u_i \right \rangle = 0$.
 
 
 
-But $$\left \langle \hat X \hat v_i \right \rangle = \left \langle \hat u_i d_i \right \rangle = d_i \left \langle \hat u_i \right \rangle $$.
+Regarding $\hat V$ and using the similar approach as above by
+computing $\left \langle \hat X^T \hat u_i \right \rangle = d_i \left
+\langle \hat v_i \right \rangle$, we have
 
-Since $$ \left \langle \hat X \hat v_i \right \rangle = 0$$, then $$d_i \left \langle \hat u_i \right \rangle = 0$$ also. Assuming that $$d_i \neq 0$$ (otherwise the variance in the exercise would just be zero), gives that $$\left \langle \hat u_i \right \rangle = 0$$.
-
-
-
-Regarding $$\hat V$$ and using the similar approach as above by computing $$\left \langle \hat X^T \hat u_i \right \rangle = d_i \left \langle \hat v_i \right \rangle$$, it seems that
-
-$$\begin{align*} <br/>\left \langle \hat X^T \hat u_i \right \rangle &= \frac{1}{N}\sum_j ( \hat X^T \hat u_i )_j \\ <br/>&= \frac{1}{N}\sum_j \sum_k x_{kj} \hat u_i(k)\\ <br/>&= \frac{1}{N}\sum_k \hat u_i(k) \sum_j x_{kj} \\<br/>&= \frac{1}{N}\sum_k \hat u_i(k) \sum_j x_{kj} \\ <br/>&= \sum_k \hat u_i(k) \left( \frac{1}{N} \sum_j x_{kj} \right)\\ <br/>\end{align*} $$
-
-We do not know anything about the sample mean over the rows of $$\hat X$$
+!bt
+\[
+\begin{align*} \left \langle \hat X^T \hat u_i \right \rangle &= \frac{1}{N}\sum_j ( \hat X^T \hat u_i )_j \\ &= \frac{1}{N}\sum_j \sum_k x_{kj} \hat u_i(k)\\  &= \frac{1}{N}\sum_k \hat u_i(k) \sum_j x_{kj} \\ &= \frac{1}{N}\sum_k \hat u_i(k) \sum_j x_{kj} \\  &= \sum_k \hat u_i(k) \left( \frac{1}{N} \sum_j x_{kj} \right)\\ 
+\end{align*}
+!et

doc/src/Regression/Regression.do.txt

@@ -1578,6 +1578,42 @@ We will come back to more interpreations after we have gone through some of the
 For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
 Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
 
+!split
+=====  Some simple codes for the SVD =====
+
+!bc pycod
+import numpy as np
+# SVD inversion
+def SVDinv(A):
+    ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
+    SVD is numerically more stable than the inversion algorithms provided by
+    numpy and scipy.linalg at the cost of being slower.
+    '''
+    U, s, VT = np.linalg.svd(A)
+    print(U)
+    print(s)
+    print(VT)
+    D = np.zeros((len(U),len(VT)))
+    for i in range(0,len(VT)):
+        D[i,i]=s[i]
+    UT = np.transpose(U); V = np.transpose(VT); invD = np.linalg.inv(D)
+    return np.matmul(V,np.matmul(invD,UT))
+
+
+X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
+print(X)
+A = np.transpose(X) @ X
+print(A)
+# Brute force inversion of super-collinear matrix
+#B = np.linalg.inv(A)
+#print(B)
+C = SVDinv(A)
+print(C)
+
+!ec
+
+The matrix $\bm{X}$ has columns that are linearly dependent. The column is the row-wise sum of the other two columns. The rank of a matrix (the column rank) is the dimension of space spanned by the column vectors. The rank of the matrix is the number of linearly independent columns, in this case just $2$. We see this from the singular values when running the above code. Running the standard inversion algorithm for matrix inversion with $\bm{X}^T\bm{X}$ results in the program terminating due to a singular matrix. 
+
 !split
 ===== Where are we going? =====