Course notes

Notes

Writing and summarizing helps me process and remember. Here, I collect longer write-ups, that are created mostly as I work through a course.

International Programme on AI Evaluation · ongoing

01 Introduction to AI Evaluation 2 lectures
AI Evaluation as a Scientific Discipline
by José Hernández-Orallo, Leverhulme Centre for the Future of Intelligence, University of Cambridge
3 Feb 2026
Introduction to AI Evaluation (Part 2)
by Jin-Dong Wang, William & Mary, Department of Data Science
8 Feb 2026
03 Metrics and Experimental Methodology 5 lectures
Statistical Foundation of AI Evaluation
by Line Clemmensen, Technical University of Denmark (DTU)
10 Feb 2026
Calibration
by Peter Flach, University of Bristol
11 Feb 2026
Experiment Design
by Line Clemmensen, Technical University of Denmark (DTU)
12 Feb 2026
ML Model Deployment and Monitoring
by Cèsar Ferri, Universitat Politècnica de València (UPV)
12 Feb 2026
Uncertainty Quantification
by Thomas Dietterich, Oregon State University
24 Feb 2026
04 Benchmarks, Leaderboards and Competitions 5 lectures
The Science of Benchmarking
by Joaquin Vanschoren, Eindhoven University of Technology
17 Feb 2026
New Benchmarking Paradigms
by Lorenzo Pacchiardi, University of Cambridge
17 Feb 2026
Saturation & Contamination
by Lorenzo Pacchiardi, University of Cambridge
17 Feb 2026
Scaling Laws
by Manuel Cebrián, Spanish National Research Council (CSIC)
24 Feb 2026
Evaluating Multi-Agent AI Systems
by Joel Z Leibo, Google DeepMind
27 Feb 2026
05 Adversarial Evaluations 4 lectures
Uplifting Evaluations & Strategic Red Teaming
by Cozmin Ududec, UK AI Security Institute
3 Mar 2026
Sociotechnical Approach to Red Teaming
by Laura Weidinger, Google DeepMind
3 Mar 2026
5 Mar 2026
Jailbreaks
by Mohammad Taufeeque, FAR.AI
5 Mar 2026
06 Construct-Based Evaluation 4 lectures
Construct-Oriented Evaluation
by Liming Jiang, Microsoft Research Asia
6 Mar 2026
Animal Psychology
by Lucy Cheke, University of Cambridge
11 Mar 2026
Robust Evaluation of Generative AI
by Marko Tešić, UK Department for Science, Innovation and Technology (DSIT)
12 Mar 2026
Measurement Theory for AI Evaluation
by Sanmi Koyejo, Stanford University
18 Mar 2026
07 Interpretability 2 lectures
ARENA, Mech Interp & SAEs
by Callum McDougall, ARENA
20 Mar 2026
Probing Neural Networks
by Fazl Barez, University of Oxford
21 Mar 2026
08 Agentic, Alignment and Control Evaluations 6 lectures
AI Control
by Tyler Tracy, Redwood Research
24 Mar 2026
AI Meets Values (I)
by Xiaoyuan Yi, Microsoft Research Asia
24 Mar 2026
Agentic Evaluation
by Cozmin Ududec, UK AI Security Institute
25 Mar 2026
AI meets Values (II)
by Xiaoyuan Yi, Microsoft Research Asia
25 Mar 2026
Deception
by Jérémy Scheurer, Apollo Research
26 Mar 2026
AI meets Values (III)
by Xiaoyuan Yi, Microsoft Research Asia
26 Mar 2026
09 Real-World Evaluation: Societal Impacts of AI 6 lectures
Human-AI Thought Partnerships
by Katie Collins, MIT (Computational Cognitive Science) & Princeton AI Lab
7 Apr 2026
Future of Work & the Economy (II)
by Jonathan Prunty, Leverhulme Centre for the Future of Intelligence, University of Cambridge
7 Apr 2026
Future of Work & the Economy (I)
by Marko Tešić, UK Department for Science, Innovation and Technology (DSIT)
7 Apr 2026
Sociotechnical Evaluation of AI Systems
by Laura Weidinger, Google DeepMind
8 Apr 2026
Fairness in AI Systems
by Angelina Wang, Cornell Tech & Cornell University
9 Apr 2026
21 Apr 2026
10 Governance, Policy and Regulation 5 lectures
AI Evaluations in Governance and Policy
by Patricia Paskov, RAND Corporation & Oxford Martin AI Governance Initiative
14 Apr 2026
Global AI Governance Landscape
by Seán Ó hÉigeartaigh, Leverhulme Centre for the Future of Intelligence, University of Cambridge
14 Apr 2026
Soft Governance
by Seán Ó hÉigeartaigh, Leverhulme Centre for the Future of Intelligence, University of Cambridge
16 Apr 2026
22 Apr 2026
Meta-Evaluation
by Patricia Paskov, RAND Corporation & Oxford Martin AI Governance Initiative
23 Apr 2026