Deep Learning

Course manual 2018/2019

Course content

Deep learning is primarily a study of multi-layered neural networks, spanning over a great range of model architectures. The following content will be the studied both from theoretical (during lectures) as well as from practical (during the practicals) point of view.

Lecture 1. History of Neural Networks and Deep Learning. A recap of previous simple machine learning models, like linear regression, logistic regression, perceptrons, and how they connect to Deep Learning.

Lecture 2. Neural Network modularity. Discuss what makes neural network so flexible, so powerful and so scaleable. Since neural networks are notoriously difficult to train, we will deepen our understanding on how to optimize them both theoretically and practically.

Lecture 3. Neural Network Optimization. Discuss how to optimize neural networks. Visit state-of-the-art techniques and explain them in depth. Discuss how to find optimal hyperparameters.

Lecture 4. Convolutional Neural Networks and Transfer Learning. The driving force behind the popularity of deep learning. CNNs have their main application in image recognition, object detection, automatic text translation and speech recognition. Also, normally deep learning requires big data. For some problems big data is not available. With transfer learning one can combine different sources of data to build a more powerful model.

Lecture 5. Modern Convolutional Neural Network Architectures. Present the most popular architectures to date and go in depth on what makes them so special.

Lecture 6. Recurrent neural networks (RNN) and Graph Neural Networks. CNNs or other traditional network architectures have only feedforward operations. With RNNs we add feedback to the model, thus providing it with memory. With RNNs one can build machines that write Shakespeare, automatic caption generators, automatic music generators or even machines that dream new pictures. Further, while CNNs and RNNs are mostly about data on some form of a grid, Graph Neural Networks will also be discussed where data is more generally organized in graphs.

Lecture 7. Generative Adversarial Networks. One of the most popular generative model families out there, we are going to go deep into them and discuss their properties, and what makes them produce so crisp generations.

Lecture 8. Deep Generative Models. A first lecture on the combination of Deep Learning and generative models, like VAEs, Boltzmann Machines, and so on.

Lecture 9. Bayesian Deep Learning. Present modern attempts on marrying the bayesian framework with Deep Models. We will discuss bayesian deep learning from both theoretical and practical point of view, like Monte Carlo Dropout Networks.

Lecture 10. Deep Sequential Models. Beyond RNNs, we will study modern attempts for deep networks that model data sequentially. Special emphasis will be given to Deep Autoregressive models.

Lecture 11. Deep Reinforcement Learning. We will discuss how can one leverage deep networks for implementing reinforcement learning agents. Different families of deep reinforcement learning algorithms will be presented, including value-based, policy-based and model-based deep reinforcement learning.

Lecture 12-14. Deep Learning today. Invited lectures, presenting new papers, presenting lecturers's own work.

Study materials

Literature

• Papers discussed in the lectures

Syllabus

• https://www.deeplearningbook.org/

• uvadlc.github.io

Practical training material

• https://github.com/uvadlc/uvadlc_practicals_2018/tree/master/pytorch

Software

• https://pytorch.org/

Other

• See also the website: http://deeplearning.net/tutorial/

Objectives

The course will study the theory of deep learning, namely of modern, multi-layered neural networks trained on big data. The course will have a slight focus on computer vision as the preferred application, although the same knowledge is applicable also to text and other data modalities. The course will be composed of a theoretical and a practical part.

During theory the students will be taught the fundamentals of deep learning and the latest, state-of-the-art models that empower popular applications, such as Google Photos, Google Translate, Google Text-to-Speech, Google Brain, Facebook Friend Finder, Self-driving cars, etc. During the theory part students will learn to re-implement similar models, as well as to build novel ones for new tasks. Also, during the theory sessions we plan to have experts from the field presenting their views on the subject.

During the practicals the students will implement from A to Z the core versions of some of the aforementioned applications. The course will focus on state-of-the-art programming frameworks. Students will learn what are the most relevant and frequent practical problems, and how they can be addressed in practice. During the projects, the students will further be able to build real-life, interesting and perhaps even marketable applications. Hopefully and depending on the student’s motivation, the applications could lead to demos that can be publicly presented.

Teaching methods

• Lectures. During the lectures students must bring their laptops, as the lectures could be interactive (to some extent).
• Practicals. During the practicals students will apply the theory to simple problems.

Learning activities

Attendance

The programme does not have requirements concerning attendance (OER-B).

Assessment

Inspection of assessed work

The manner of inspection will be communicated via the digitial learning environment.

Assignments

Assignment 1: MLPs, CNNs and Backpropagation

Assignment 2: Recurrent Neural Networks

Assignment 3: Deep Generative Models

To maximize the learning outcome, all assignments will be done individually.

Fraud and plagiarism

The 'Regulations governing fraud and plagiarism for UvA students' applies to this course. This will be monitored carefully. Upon suspicion of fraud or plagiarism the Examinations Board of the programme will be informed. For the 'Regulations governing fraud and plagiarism for UvA students' see: www.student.uva.nl

Course structure

Timetable

The schedule for this course is published on DataNose.

Additional information

Maximum number of students: 24

Recommended prior knowledge:

• As the course runs for only two months, students should be prepared with some background knowledge.
• Good math knowledge. Linear Algebra, Calculus (Derivatives, chain rule, etc.).
• Good programming knowledge: Python

Contact information

Coordinator

• dr. E. Gavves