FYS-STK4155/doc/LectureNotes/_build/jupyter_execute/exercisesweek47.ipynb

Exercise week 47

November 18-22, 2024

Date: Deadline is Friday November 22 at midnight

Overarching aims of the exercises this week

The exercise set this week is meant as a summary of many of the central elements in various machine learning algorithms, with a slight bias towards deep learning methods and their training. You don't need to answer all questions.

The last weekly exercise (week 48) is a general course survey.

Exercise 1: Linear and logistic regression methods

1. What is the main difference between ordinary least squares and Ridge regression?

2. Which kind of data set would you use logistic regression for?

3. In linear regression you assume that your output is described by a continuous non-stochastic function f(x). Which is the equivalent function in logistic regression?

4. Can you find an analytic solution to a logistic regression type of problem?

5. What kind of cost function would you use in logistic regression?

Exercise 2: Deep learning

1. What is an activation function and discuss the use of an activation function? Explain three different types of activation functions?

2. Describe the architecture of a typical feed forward Neural Network (NN).

3. You are using a deep neural network for a prediction task. After training your model, you notice that it is strongly overfitting the training set and that the performance on the test isn’t good. What can you do to reduce overfitting?

4. How would you know if your model is suffering from the problem of exploding Gradients?

5. Can you name and explain a few hyperparameters used for training a neural network?

6. Describe the architecture of a typical Convolutional Neural Network (CNN)

7. What is the vanishing gradient problem in Neural Networks and how to fix it?

8. When it comes to training an artificial neural network, what could the reason be for why the cost/loss doesn't decrease in a few epochs?

9. How does L1/L2 regularization affect a neural network?

10. What is(are) the advantage(s) of deep learning over traditional methods like linear regression or logistic regression?

Exercise 3: Decision trees and ensemble methods

1. Mention some pros and cons when using decision trees

2. How do we grow a tree? And which are the main parameters?

3. Mention some of the benefits with using ensemble methods (like bagging, random forests and boosting methods)?

4. Why would you prefer a random forest instead of using Bagging to grow a forest?

5. What is the basic philosophy behind boosting methods?

Exercise 4: Optimization part

1. Which is the basic mathematical root-finding method behind essentially all gradient descent approaches(stochastic and non-stochastic)?

2. And why don't we use it? Or stated differently, why do we introduce the learning rate as a parameter?

3. What might happen if you set the momentum hyperparameter too close to 1 (e.g., 0.9999) when using an optimizer for the learning rate?

4. Why should we use stochastic gradient descent instead of plain gradient descent?

5. Which parameters would you need to tune when use a stochastic gradient descent approach?

Exercise 5: Analysis of results

1. How do you assess overfitting and underfitting?

2. Why do we divide the data in test and train and/or eventually validation sets?

3. Why would you use resampling methods in the data analysis? Mention some widely popular resampling methods.