CS7DS2 Assignment Brief
1. It is thought that the “noise” added to the gradient when using mini-batch SGD acts as a regulariser and so helps prevent overfitting. Your task is to write a short report critically evaluating the use of mini-batch SGD to reduce overfitting and improve generalisation performance.
The choice of model and dataset to use is up to you, but you need to justify why your choice is appropriate – in particular you should take care to make sure that overfitting takes place when using gradient descent (or SGD with a large batch size). Its also probably a good idea to look at two models/datasets so that you can compare/contrast them. You should investigate the role of mini-batch size on overfitting (remember that SGD becomes gradient descent the batch size equals the full dataset), and also the interplay between step size and mini-batch size. You should also investigate the use of constant-step size vs adaptive approaches such as adam.
Be sure to split the data into training, test and validation sets (cross-validation does not measure generalisation behaviour adequately). Also bear in mind the random nature of SGD, which means that you will probably need to do multiple runs and look at both the average behaviour and the fluctuations about the average from run to run.
Remember that most of the marks are for critical analysis/discussion.
indicative breakdown
(i) methodology 30 marks,
(ii) evaluation and critical discussion 45 marks,
(iii) report organisation and presentation 10 marks.
2. (a) Give short code for mini-batch SGD and explain its operation. [5 marks]
(b) Explain how a projected gradient descent can be used to enforce a constraint on the decision variables. Illustrate with a brief example. [5 marks]
(c) Briefly describe the RMSProp algorithm. Discuss how it differs from Adagrad (and why), and how it can change both the magnitude and direction of the step taken. [5 marks]
Total Marks : 85 marks