Big Data

Course manual 2024/2025

Course content

Data is at the center of modern businesses and institutions, and we are experiencing a big shift towards predictive, data-driven decision making in recent years. This development has given rise to Big Data, a novel set of data storage and processing methods, accompanied by a new software stack that serves as the foundation for the ongoing AI revolution. Pioneered by companies building web-scale search engines such as Google, Big Data technologies are used across all industries nowadays and are a major component of modern cloud infrastructure.

In this course, we study various abstractions, processing techniques and Big Data systems for working with large collections of data, including relational database management systems, MapReduce and Apache Spark. We address the foundations of Big Data applications, as well as topics like choosing a suitable data representation and implementing distributed processing operations. We review and focus on the foundations of distributed data processing and programming models for parallelisable programs, as well as on data quality and responsible data management.

You will learn theoretical concepts for data processing during the lectures. In the lab you will gain hands-on experience through a number of coding assignments and by participating in a Kaggle-like Big Data competition. In addition, we offer online tutorials for hands-on experience with established industry software packages. Finally, experts from the field (both academic as well as industry colleagues) are invited for giving guest lectures.

Study materials

Literature

• Book Chapters

• Scientific Papers

• Presentation Slides

Syllabus

• A syllabus containing articles and chapters will be made available at the beginning of the course. 

Practical training material

• Programming assignments

• Kaggle-like data project

• Video Tutorials

Software

• Python

• Jupyter

• DuckDB

• PySpark

Other

• Videos

Objectives

• Explain the high impact and potential of big data technology
• Create a scalable Big Data application for a given scenario
• Analyse data schemas and distributed data processing strategies
• Design a scalable Big Data application for a machine learning task and present findings in a poster session
• Explain what relates and differentiates relational data processing, MapReduce and Resilient Distributed Datasets
• Describe common data quality issues and apply error detection and data cleaning methods
• Program key Big Data systems

Teaching methods

• Lecture
• Self-study
• Computer lab session/practical training
• Presentation/symposium
• Working independently on e.g. a project or thesis

Learning activities

Attendance

In TER part B of this programme no requirements regarding attendance are mentioned.

Additional requirements for this course:

Participation will be measured. Attendance in the lab sessions is highly recommended in order to attain the programming skills and background required for the assignments and the project.

Assessment

To pass the course, all parts should be passed. The exam will be an open book exam (based on the materials provided on canvas), and, in case of a resit, the resit grade will count.

Assignments

Programming assignments 1 & 2

• Individual assignments, will be auto-graded with immediate feedback.

Open questions

• Individual assignment, will be graded on canvas.

Project poster presentation

• Group assignment, will be graded via a poster session.

Fraud and plagiarism

The 'Regulations governing fraud and plagiarism for UvA students' applies to this course. This will be monitored carefully. Upon suspicion of fraud or plagiarism the Examinations Board of the programme will be informed. For the 'Regulations governing fraud and plagiarism for UvA students' see: www.student.uva.nl

Course structure

Additional information

The course will be taught in English. 

Basic programming skills, knowledge of computing systems and machine learning are required. 

Contact information

Coordinator

• dr. H. Harmouch