Information Theory

Course manual 2025/2026

Course content

Information theory was developed by Claude E. Shannon in the 1950s to investigate the fundamental limits on signal-processing operations such as compressing data and on reliably storing and communicating data. These tasks have turned out to be fundamental for all of computer science.

In this course, we quickly review the basics of probability theory and introduce concepts such as (conditional) Shannon entropy, mutual information and entropy diagrams. Then, we prove Shannon's theorems about data compression and channel coding. An interesting connection with graph theory is made in the setting of zero-error information theory. We also cover some aspects of information-theoretic security such as perfectly secure encryption.

Study materials

Literature

• Good reference: [CF] Ronald Cramer, Serge Fehr: 'The Mathematical Theory of Information, and Applications': http://homepages.cwi.nl/%7Eschaffne/courses/inftheory/2016/notes/CramerFehr.pdf, lecture notes, Version 2.0
• Good reference: [CT] Thomas M. Cover, Joy A. Thomas. 'Elements of information theory': http://onlinelibrary.wiley.com/book/10.1002/0471200611, 2nd Edition. New York: Wiley-Interscience, 2006. ISBN 0-471-24195-4.
• Good reference: [MacKay] David J. C. MacKay. 'Information Theory, Inference, and Learning Algorithms': http://www.inference.phy.cam.ac.uk/mackay/itila/book.html. Cambridge University Press, 2003. ISBN 0-521-64298-1

Other

• The material will be presented in black-boards lectures.

Objectives

• The student can apply probabilistic methods to discrete events and random variables
• The student can analyze information content and uncertainty using entropy and mutual
• The student can design and evaluate lossless compression schemes
• The student can model and assess communication channels and error-correcting systems 

Teaching methods

• Lecture
• Seminar

This is a 6 ECTS course, which comes to roughly 20 hours of work per week.

There will be homework exercises every week to be handed in one week later before the start of the exercise session. The answers must be written in LaTeX. The homework exercises will be solved in groups of 3-4 students. 

Additionally, there will be short quizzes to be solved on canvas. You are encouraged to solve these quizzes individually to test your understanding of the course material before attempting the homework problems. 

Learning activities

Attendance

This programme does not have requirements concerning attendance (TER-B).

Assessment

The final grade will be determined by the following calculation:

70% exam grade + 30% homework grade

It is additionally required that you score at least 50% on the final exam to pass the course. The same rule applies for the re-sit exam (which would replace your exam grade). 

There will be one homework problem set per week (in total 6), posted on the course homepage by Monday. You must submit your completed homework (on Canvas) before Monday the week after. We will ask you to collaborate and submit in groups of 3-4 students. The solutions will be discussed in the exercise classes (among other things). Assignments will be accepted late only if you have extenuating circumstances (such as sickness or family emergency) and provided you confirm with the lecturer before the deadline. 

Everything that we discussed in class, on the homeworks, on in the canvas modules is in principle in scope for the exam. We recommend that you revisit the homework problems, exercises, and (ungraded) canvas quizzes as preparation.

Assignments

There will be 6 homework assignments, to be completed in groups of 3-4. For each homework, you will have one week to complete it. Solutions to the homework problems will be provided in the exercise sessions upon request.

Fraud and plagiarism

The 'Regulations governing fraud and plagiarism for UvA students' applies to this course. This will be monitored carefully. Upon suspicion of fraud or plagiarism the Examinations Board of the programme will be informed. For the 'Regulations governing fraud and plagiarism for UvA students' see: www.student.uva.nl

Course structure

Additional information

Familiarity with probability theory will be useful, but is not explicitly required. No programming will be required for this course.

Contact information

Coordinator

• Nicolas Resch

Staff

• Daan Planken
• Lydia Tasiou
• Sebastian Verschoor