-
@@ -284,21 +284,21 @@ analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters (estimators) \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) -$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +$$ MSE(\boldsymbol{y},\tilde{\boldsymbol{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$
-and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +and the \( R^2 \) score function. If \( \tilde{\boldsymbol{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as $$ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$
-where we have defined the mean value of \( \hat{y} \) as +where we have defined the mean value of \( \boldsymbol{y} \) as $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -406,7 +406,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed.
Implement the \( k \)-fold cross-validation algorithm (write your own diff --git a/doc/Projects/2021/Project1/html/Project1-bs.html b/doc/Projects/2021/Project1/html/Project1-bs.html index 46957ed0a..8fe85fccd 100644 --- a/doc/Projects/2021/Project1/html/Project1-bs.html +++ b/doc/Projects/2021/Project1/html/Project1-bs.html @@ -168,7 +168,7 @@ MathJax.Hub.Config({
-
@@ -284,21 +284,21 @@ analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters (estimators) \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) -$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +$$ MSE(\boldsymbol{y},\tilde{\boldsymbol{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$
-and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +and the \( R^2 \) score function. If \( \tilde{\boldsymbol{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as $$ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$
-where we have defined the mean value of \( \hat{y} \) as +where we have defined the mean value of \( \boldsymbol{y} \) as $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -406,7 +406,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed.
Implement the \( k \)-fold cross-validation algorithm (write your own diff --git a/doc/Projects/2021/Project1/html/Project1.html b/doc/Projects/2021/Project1/html/Project1.html index 545df6ab9..405e39681 100644 --- a/doc/Projects/2021/Project1/html/Project1.html +++ b/doc/Projects/2021/Project1/html/Project1.html @@ -126,7 +126,7 @@ MathJax.Hub.Config({
-
-and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +and the \( R^2 \) score function. If \( \tilde{\boldsymbol{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as $$ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$
-where we have defined the mean value of \( \hat{y} \) as +where we have defined the mean value of \( \boldsymbol{y} \) as $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -362,7 +362,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed.
Implement the \( k \)-fold cross-validation algorithm (write your own diff --git a/doc/Projects/2021/Project1/ipynb/Project1.ipynb b/doc/Projects/2021/Project1/ipynb/Project1.ipynb index b32c902df..7bd009df3 100644 --- a/doc/Projects/2021/Project1/ipynb/Project1.ipynb +++ b/doc/Projects/2021/Project1/ipynb/Project1.ipynb @@ -10,7 +10,7 @@ " \n", "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, University of Oslo, Norway\n", "\n", - "Date: **Sep 14, 2021**\n", + "Date: **Oct 4, 2021**\n", "\n", "## Regression analysis and resampling methods\n", "\n", @@ -149,7 +149,7 @@ "metadata": {}, "source": [ "$$\n", - "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "MSE(\\boldsymbol{y},\\tilde{\\boldsymbol{y}}) = \\frac{1}{n}\n", "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", "$$" ] @@ -158,7 +158,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "and the $R^2$ score function. If $\\tilde{\\hat{y}}_i$ is the predicted\n", + "and the $R^2$ score function. If $\\tilde{\\boldsymbol{y}}_i$ is the predicted\n", "value of the $i-th$ sample and $y_i$ is the corresponding true value,\n", "then the score $R^2$ is defined as" ] @@ -168,7 +168,7 @@ "metadata": {}, "source": [ "$$\n", - "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", "$$" ] }, @@ -176,7 +176,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where we have defined the mean value of $\\hat{y}$ as" + "where we have defined the mean value of $\\boldsymbol{y}$ as" ] }, { @@ -200,6 +200,7 @@ "sure you have installed it). This function is called\n", "$train\\_test\\_split$. **You should present a critical discussion of why and how you have scaled or not scaled the data**.\n", "\n", + "\n", "It is normal in essentially all Machine Learning studies to split the\n", "data in a training set and a test set (eventually also an additional\n", "validation set). There\n", @@ -311,7 +312,7 @@ "The aim here is to write your own code for another widely popular\n", "resampling technique, the so-called cross-validation method. Again,\n", "before you start with cross-validation approach, you should scale your\n", - "data.\n", + "data if you think this is needed.\n", "\n", "Implement the $k$-fold cross-validation algorithm (write your own\n", "code) and evaluate again the MSE function resulting\n", diff --git a/doc/Projects/2021/Project1/ipynb/ipynb-Project1-src.tar.gz b/doc/Projects/2021/Project1/ipynb/ipynb-Project1-src.tar.gz index 3d71f4bd3..36579028d 100644 Binary files a/doc/Projects/2021/Project1/ipynb/ipynb-Project1-src.tar.gz and b/doc/Projects/2021/Project1/ipynb/ipynb-Project1-src.tar.gz differ diff --git a/doc/Projects/2021/Project1/pdf/Project1.p.tex b/doc/Projects/2021/Project1/pdf/Project1.p.tex index a544ac850..2654c2293 100644 --- a/doc/Projects/2021/Project1/pdf/Project1.p.tex +++ b/doc/Projects/2021/Project1/pdf/Project1.p.tex @@ -132,7 +132,7 @@ Project 1 on Machine Learning, deadline October 4, 2021 % --- begin date --- \begin{center} -Sep 14, 2021 +Oct 4, 2021 \end{center} % --- end date --- @@ -242,19 +242,19 @@ analysis using polynomials in $x$ and $y$ up to fifth order. Find the \href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\[ MSE(\bm{y},\tilde{\bm{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \] -and the $R^2$ score function. If $\tilde{\hat{y}}_i$ is the predicted +and the $R^2$ score function. If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \] -where we have defined the mean value of $\hat{y}$ as +where we have defined the mean value of $\bm{y}$ as \[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -268,6 +268,7 @@ splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it). This function is called $train\_test\_split$. \textbf{You should present a critical discussion of why and how you have scaled or not scaled the data}. + It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional validation set). There @@ -345,7 +346,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed. Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting diff --git a/doc/Projects/2021/Project1/pdf/Project1.pdf b/doc/Projects/2021/Project1/pdf/Project1.pdf index 647a3f09f..356eefbf9 100644 Binary files a/doc/Projects/2021/Project1/pdf/Project1.pdf and b/doc/Projects/2021/Project1/pdf/Project1.pdf differ diff --git a/doc/Projects/2021/Project1/pdf/Project1.tex b/doc/Projects/2021/Project1/pdf/Project1.tex index 5d8fbc2d9..63a62b2e2 100644 --- a/doc/Projects/2021/Project1/pdf/Project1.tex +++ b/doc/Projects/2021/Project1/pdf/Project1.tex @@ -102,7 +102,7 @@ Project 1 on Machine Learning, deadline October 4, 2021 % --- begin date --- \begin{center} -Sep 14, 2021 +Oct 4, 2021 \end{center} % --- end date --- @@ -212,19 +212,19 @@ analysis using polynomials in $x$ and $y$ up to fifth order. Find the \href{{https://en.wikipedia.org/wiki/Confidence_interval}}{confidence intervals} of the parameters (estimators) $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\[ MSE(\bm{y},\tilde{\bm{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \] -and the $R^2$ score function. If $\tilde{\hat{y}}_i$ is the predicted +and the $R^2$ score function. If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \] -where we have defined the mean value of $\hat{y}$ as +where we have defined the mean value of $\bm{y}$ as \[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -238,6 +238,7 @@ splitting training data provided by the library \textbf{Scikit-Learn} (make sure you have installed it). This function is called $train\_test\_split$. \textbf{You should present a critical discussion of why and how you have scaled or not scaled the data}. + It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional validation set). There @@ -315,7 +316,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed. Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting diff --git a/doc/src/Projects/2021/Project1/Project1.do.txt b/doc/src/Projects/2021/Project1/Project1.do.txt index 5c161ece0..4d373f296 100644 --- a/doc/src/Projects/2021/Project1/Project1.do.txt +++ b/doc/src/Projects/2021/Project1/Project1.do.txt @@ -111,22 +111,22 @@ analysis using polynomials in $x$ and $y$ up to fifth order. Find the variances, evaluate the Mean Squared error (MSE) !bt -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\[ MSE(\bm{y},\tilde{\bm{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \] !et -and the $R^2$ score function. If $\tilde{\hat{y}}_i$ is the predicted +and the $R^2$ score function. If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as !bt \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \] !et -where we have defined the mean value of $\hat{y}$ as +where we have defined the mean value of $\bm{y}$ as !bt \[ @@ -142,6 +142,7 @@ splitting training data provided by the library _Scikit-Learn_ (make sure you have installed it). This function is called $train\_test\_split$. _You should present a critical discussion of why and how you have scaled or not scaled the data_. + It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (eventually also an additional validation set). There @@ -228,7 +229,7 @@ Note also that when you calculate the bias, in all applications you don't know t The aim here is to write your own code for another widely popular resampling technique, the so-called cross-validation method. Again, before you start with cross-validation approach, you should scale your -data. +data if you think this is needed. Implement the $k$-fold cross-validation algorithm (write your own code) and evaluate again the MSE function resulting diff --git a/doc/src/week37/programs/LinearReg.py b/doc/src/week37/programs/LinearReg.py new file mode 100644 index 000000000..be84e77f5 --- /dev/null +++ b/doc/src/week37/programs/LinearReg.py @@ -0,0 +1,270 @@ +import numpy as np +from sklearn import linear_model, metrics +from sklearn.model_selection import train_test_split + +class linregOwn: + """ + A class of linear regressions. Perform ordinarly least squares (OLS) and Ridge regression manually. Lasso + is performed using scikit-learn functionality. + """ + def __init__(self, method = 'ols'): + """ + Constructor + + Determines the method used in the fitting + + Arguments: + method: string type. Accepts either 'ols', 'ridge' or 'lasso'. + + """ + self.method = method + self.yHat = None + self.X = None + self.y = None + self.beta = None + + self._MSE = None + self._R2 = None + self._betaVariance = None + self.lambda_ = None + + def fit(self, X_train, y_train, lambda_ = 0): + """ + Performs the fit of OLS, Ridge or Lasso, depending on the argument provided initially. + + Arguments: + X_train: Covariate matrix of the train data set, i.e. design matrix of + the shape m x p where m is the number of rows and p is the number of columns + (i.e. p is the complexity parameter). + y_train: Outcome variable, 1D numpy array + lambda_: float type. Shrinkage parameter for ridge and lasso methods. The higher value, higher shrinkage. + lambda_ is set to 0 for the OLS regression + + """ + self.X_train = X_train + self.y_train = y_train + self.lambda_ = lambda_ + if self.method == 'ols': + self._olsFit(X_train, y_train) + if self.method == 'ridge': + self._ridgeFit(X_train, y_train, lambda_) + if self.method == 'lasso': + self._lassoFitSKL(X_train, y_train, lambda_) + return self.beta + + def _olsFit(self, X_train, y_train): + """ + Performs the ordinary least squares (OLS) fit on the provided data using singular value decomposition(SVD). + + + Arguments: + + X_train: Covariate matrix of the train data set, i.e. design matrix of + the shape m x p where m is the number of rows and p is the number of columns + (i.e. p is the complexity parameter). + y_train: Outcome variable, 1D numpy array + + Returns: + beta : numpy.array + The beta parameters from the performed fit + """ + self.X_train = X_train + self.y_test = y_train + U, S, VT = np.linalg.svd(self.X_train, full_matrices=True) + S_inverse = np.zeros(shape=self.X_train.shape) + ##S is a vector, with shape of the number of columns + S_inverse[:S.shape[0], :S.shape[0]] = np.diag(1/S) + self.beta = np.dot(VT.T, np.dot(S_inverse.T, np.dot(U.T, self.y_train))) + #self.beta = np.linalg.inv(np.dot(X.T,X)).dot(X.T, y) + + def _ridgeFit(self, X_train, y_train, lambda_): + """ + Performs the ridge regression fit + + Arguments: + X_train: Covariate matrix of the train data set, design matrix of + the shape m x p (m_train_rows, p_columns). + y_train: Outcome variable, 1D numpy array, dimension m x 1 + lambda_: Integer type. The shrinkage parameter + + Returns: + beta : numpy.array + The beta parameters from the performed fit + """ + self.X_train = X_train + self.y_train = y_train + self.lambda_ = lambda_ + self.beta = np.dot(np.linalg.inv(np.dot(X_train.T,X_train) + self.lambda_ * np.eye(X_train.shape[1])), np.dot(X_train.T,y_train)) + + def _lassoFitSKL(self, X_train, y_train, lambda_): + """ + Performs lasso fit using scikit-learn functionality. + + Arguments: + X_train: Covariate matrix of the train data set, design matrix of + the shape m x p (m_train_datapoints, p_parameters). + y_train: Outcome variable, 1D numpy array, dimension m x 1 + lambda_: Integer type. The shrinkage parameter + + Returns: + self.beta : numpy.array + The beta parameters from the performed fit + """ + self.regression = linear_model.Lasso(fit_intercept=True, max_iter=1000000, alpha=self.lambda_) + self.regression.fit(X_train,y_train) + self.beta = self.regression.coef_ + self.beta[0] = self.regression.intercept_ + + def predict(self, X_test): + """ + Performs prediction of the fitted model on the provided test data set. + + Arguments: + X_test: Design matrix, covariate matrix, dimension k x p (k_test_rows, p_columns) + + Returns: self.yHat + numpy 1D array, prediction values of dimension k x p + """ + self.X_test = X_test + self._predictOwntest(X_test) + return self.yHat + + def _predictOwntest(self, X_test): + """ + Performs manual prediction of the given model on the train data. + """ + self.X_test = X_test + self.yHat = np.dot(self.X_test, self.beta) + + def MSE(self, y_test): + """ + Calculates the mean squared error (MSE) manually after the fit and prediction have been implemented. + + Arguments: + y_test: Outcome variable, 1D numpy array, dimension k x 1 (k_test_rows, 1_column) + + Returns: self._MSE + The mean squared error of the predicted model + """ + self.y_test = y_test + if self.yHat is None : + self._predictOwntest(X_test) + N = self.yHat.size + self._MSE = (np.sum((self.y_test - self.yHat)**2))/N + return self._MSE + + def R2(self, y_test): + """ + Calculates R2 score manually after the fit and prediction have been implemented. + + Arguments: + y_test: Outcome variable, 1D numpy array, dimension k x 1 (k_test_rows, 1_column) + + Returns: self._R2 + The R2 score of the predicted model + """ + self.y_test = y_test + if self.yHat is None: + self._predictOwntest(X_test) + yMean = (1.0 / self.y_test.size) * np.sum(self.y_test) + self._R2 = 1.0 - np.sum((self.y_test - self.yHat)**2) / np.sum((self.y_test - yMean)**2) + return self._R2 + + def CI(self, y_test): + """ + Calculates confidence intervals manually after the fit and prediction have been implemented. + + Arguments: + y_test: Outcome variable, 1D numpy array, dimension k x 1 (k_test_rows, 1_column) + + Returns: var, Lower, Upper + Variance, Lower and Upper bounds of the confidence intervals for the parameter self.beta + """ + self.y_test = y_test + if self.yHat is None: + self._predictOwntest(X_test) + sigma2 = np.sum(((self.y_test - self.yHat)**2))/(self.y_test.size - self.beta.size) + var = np.diag(np.linalg.inv(np.dot(self.X_test.T, self.X_test))) * sigma2 + Lower = self.beta - 1.96*np.sqrt(var) + Upper = self.beta + 1.96*np.sqrt(var) + return var, Lower, Upper + + + + + + ###Implementation through scikitlearn +class linregSKL: + def __init__(self, method = 'ols'): + """ + A class of linear regressions. Perform ordinarly least squares (OLS) and Ridge and Lasso + using scikit-learn functionality. + + """ + self.method = method + self.yHat = None + self.X = None + self.y = None + self.beta = None + + self._MSE = None + self._R2 = None + self._betaVariance = None + + + def fit(self, X_train, y_train, lambda_ = 0): + self.X_train = X_train + self.y_train = y_train + if self.method == 'ols': + self._olsSKLfit(X_train, y_train) + if self.method == 'ridge': + self._sklRidgeFit(X_train, y_train, lambda_) + if self.method == 'lasso': + self._SKLlassoFit(X_train, y_train, lambda_) + return self.beta + + def _olsSKLfit(self, X_train, y_train): + self.X_train = X_train + self.y_train = y_train + ##We already have standardized data from design matrix + self.ols = linear_model.LinearRegression().fit(self.X_train, self.y_train) + self.beta = self.ols.coef_ + self.beta[0] = self.ols.intercept_ + + def _SKLlassoFit(self, X_train, y_train, lambda_): + self.regression = linear_model.Lasso(fit_intercept=True, max_iter=100000, alpha=self.lambda_) + self.regression.fit(X_train,y_train) + self.beta = self.regression.coef_ + self.beta[0] = self.regression.intercept_ + + def _sklRidgeFit(self, X_train, y_train, lambda_): + self.regression = linear_model.Ridge(fit_intercept=True, alpha=self.lambda_) + self.regression.fit(X,y) + self.beta = self.regression.coef_ + self.beta[0] = self.regression.intercept_ + + def predict(self, X_test): + self.X_test = X_test + if self.method == 'ols': + self._sklPredict(X_test) + return self.yHat + + def _sklPredict(self, X_test): + self.X_test = X_test + ## Since our data contains 1-s, we should subtract intercept, since scikit learn additionally + ##generates the 1-s + self.yHat = self.ols.predict(self.X_test) - self.beta[0] + + def MSE(self, y_test): + self.y_test = y_test + if self.yHat is None : + self._sklPredict(X_test) + self._MSE = metrics.mean_squared_error(self.y_test, self.yHat) + return self._MSE + + def R2(self, y_test): + self.y_test = y_test + if self.yHat is None : + self._sklPredict() + self._R2 = metrics.r2_score(self.y_test, self.yHat) + return self._R2 \ No newline at end of file