NOTE : The functions that are to be implemented in the following, must be written in vectorized form. That is, it must be able to handle data of any size or shape.

1. Complete the following functions in linearRegression/linear_regression.py

• fit_sklearn_LR - solve the linear regression problem by calling the linear regression solver in scikit-learn [0.5 marks]

• fit_normal_equations - solve the linear regression problem by using the closed form solution of the normal equations [0.5 marks]

• fit_SVD - solve the linear regression problem by computing the SVD of the data matrix. [0.5 marks]

• mse_loss - compute the mean squared error of the predicted solution

• predict - compute the prediction for testing data

  For the second and third part, you can call the respective functions from numpy.linalg library. Report the errors by running the code in Q1_test.py and compare the time taken for each technique. [1.5 marks]

2. Complete the following functions in linearRegression/linear_regression.py

• compute_gradient - compute the analytical gradient of the unregularized mse_loss and mse_loss with ridge regularization [0.5 marks]

• compute_jax_gradient - compute the gradient of the unregularized mse_loss, mse_loss with LASSO regularization and mse_loss with ridge regularization, using JAX [0.5 marks]

• fit_gradient_descent - implement the batch gradient descent algorithm [1 marks]

• fit_SGD_with_momentum - implement the SGD algorithm with momentum [0.5 marks]

  In Q2_test.py,  solve the regression problem by calling the following functions

• fit_gradient_descent using the manually computed gradients for each of unregularized mse_loss, and mse_loss with ridge regularization

• fit_gradient_descent using the JAX gradients for each of unregularized mse_loss, mse_loss with LASSO regularization and mse_loss with ridge regression

• fit_gradient_descent for running SGD on mse_loss with ridge regularization

• fit_gradient_descent for running minibatch SGD on mse_loss with ridge regularization

• fit_SGD_with_momentum for running SGD on mse_loss with ridge regularization

  Report the errors for each of these methods as well as the time taken for each, by editing the code in Q2_test.py. For LASSO regression, you need to implement only the JAX gradients. Try out a variety of learning rates, 'lambda's  and 'momentums', and  report the errors and times in a tabular form. Which hyperparameters give the best result?(Resources : SGD, Momentum)[2.5 marks]

3. For gradient descent (any of the above implementations) create a gif/video of gradient descent updates showing how the line fit improves over the iterations. Also, show the gradient descent updates on a contour plot as well as the loss surface. These can be done by completing the functions plot_line_fit, plot_surface, and plot_contour in linearRegression/linear_regression.py. The contour and surface must be plotted for the parameters theta_0 and theta_1 (c and m  for the linear regression fit) [0.5 each] The gifs need to show only the first 10 iterations. You can create these gifs using any online services after you plot individual images for each gradient update. Alternatively you can also use the matplotlib.animation library. The plots can be generated by running the code in Q3_test.py. [1.5 marks]

4. Write a function inspired by sklearn's polynomial preprocessing: (https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html) your function should have: degree and include bias parameters only. For this assignment, assume that input is a 1-dimensional numpy array. For example, if an input sample is np.array([a, b]), the degree-2 polynomial features with "include_bias=True" are [1, a, b, a², b²].Edit preprocessing/polynomial_features . Now, using your newly created function for polynomial feature generation above, plot norm of theta v/s degree when you fit linear regression using the polynomial of degree d. What can you conclude? Edit Q4_test.py [0.5 marks]

5. Use the above code and for degree = 1, 3, 5, 7, 9, plot magnitude of  the norm of parameter vector, theta v/s degree for varying N, where N is the size of the data set (size of x, and thus y). What can you conclude? Edit Q5_plot.py [0.5 marks]

6. Using StandardScaler() from sklearn, run any of the regression algorithms above. Use the diabetes dataset, available in sklearn for the regression problem. Note that you have to set scaled=False. Also,report the errors with and without normalizing. Edit Q6_input_normalize.py [0.5 marks]

7. Consider the Minimum Daily Temperatures dataset from Australia. This is a dataset for a forecasting task. That is, given temperatures up to date T, design a forecasting (autoregressive) model to predict the temperature on date T+k (for some k > 0). You can refer to link 1, link 2 for more information on autoregressive models. Apply your favorite regression technique defined in the previous questions. Plot the fit of your predictions vs the true values and report the RMSE. Edit Q7_forecast.py [1 marks]