Introduction to Optimization for Deep Learning

Fall 2025 (Third year PhD)

Context

During fall 2025, I gave an 18h course on optimization methods for deep learning. Students were last year Master students.

Content of the course

This course aimed at giving the student both a theoritical background on deep learning and practical experience for solving simple problems. It was focused on optimization principles in deep neural networks (gradient decent, backpropagation…), on most common optimizers (ADAM, RMSProp, AdaGrad…) and on various neural structures (MLP, U-Net, GANs, …). During the three practical sessions, the students could implement their own neural networks, first from scratch and then using PyTorch and solve some standard problems (MNIST classification, Image generation, …)