In the ordinary least squares method we choose the cost function $$ \begin{align} C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. \tag{6} \end{align} $$
We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. This yields the expression for \( \boldsymbol{\beta} \) to be $$ \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, $$
which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an intercept, i.e., a constant term, we must make sure that the first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here
X_train_own = np.concatenate(
(np.ones(len(X_train))[:, np.newaxis], X_train),
axis=1
)
X_test_own = np.concatenate(
(np.ones(len(X_test))[:, np.newaxis], X_test),
axis=1
)
def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
return scl.inv(x.T @ x) @ (x.T @ y)
beta = ols_inv(X_train_own, y_train)