Information Theoretic Learning

Course manual 2025/2026

Course content

The Minimum Description Length principle is a philosophy of learning that is slightly different from common approaches such as Bayesian inference, classical statistical methods, or other machine learning solutions. It is based on the idea that any pattern or structure you detect in any data set can be translated into a more efficient representation of that data set: the goal is to exploit the strong link between understanding data and being able to compress data. This allows a mathematically rigorous interpretation of Occam’s Razor: if several explanations of the data are available, Occam’s razor tells us to look for the explanation that offers the best compression.

The philosophy is directly practical and can be applied to statistical questions, such as model selection, parameter estimation, and prediction tasks. Should we choose a complex model with many parameters, or a simpler model that is more easily trained? In this course you will learn to use MDL to solve practical problems, which also provides a new perspective on classical statistical issues.

The course borrows ideas and insights from many different fields, and a major theme is to go into the connections between such diverse fields as information theory, Bayesian statistics, frequentist statistics, Kolmogorov complexity, online learning, machine learning, and finance. The course is mostly theoretical but the homework also includes programming assignments.

Study materials

Literature

• The following literature is useful but not required for doing well in this course:

  • P. Grünwald: The Minimum Description Length Principle. MIT press, 2007.
  • T. Cover and J. Thomas: Elements of Information Theory. 2nd ed. Wiley-Interscience, 2006.
  • M. Li and P. Vitanyi: An Introduction to Kolmogorov Complexity and Its Applications, 2nd ed. Springer 1997.

   

Objectives

• Identify interpretations of fundamental concepts like "information" and "probability".
• Explain the relationships between information theory, Bayesian statistics, Kolmogorov complexity, frequentist statistics and machine learning.
• Use the MDL principle to assess under which circumstances a given model will be effective.
• Apply efficient coding techniques to data analysis.

Teaching methods

• Lecture
• Seminar
• Self-study

Learning activities

Attendance

Assessment

In addition to the written exam, there will be weekly homework. Your final grade is the average of your homework grade and exam grade. Both homework and exam grade have to be >= 5 to pass the course; final grade has to be >= 5.5.

Inspection of assessed work

Homework grades can be discussed with the TA during the tutorials. The exam can be reviewed during a time window that will be indicated on Canvas when the exam grades are published.

Assignments

A set of assignments has to be submitted weekly via Canvas. Students can work alone or in pairs on these assignments.

Fraud and plagiarism

The 'Regulations governing fraud and plagiarism for UvA students' applies to this course. This will be monitored carefully. Upon suspicion of fraud or plagiarism the Examinations Board of the programme will be informed. For the 'Regulations governing fraud and plagiarism for UvA students' see: www.student.uva.nl

Course structure

Contact information

Coordinator

• dr. Steven de Rooij

Teaching Assistant:

• Rafael Gomes, rafael.j.gomes23@gmail.com