Advanced Seminar on Foundation Models

Logistics

Instructor: Vishnu Lokhande
Piazza: sign-up link
Lectures: Wed, 4:00PM - 6:40PM, Davis 113A

Description

This course will explore the cutting-edge field of foundation models, which have transformed contemporary machine learning on a wide range of tasks after being trained on enormous volumes of raw data. We will examine the fundamental ideas that underpin the effectiveness or shortcomings of these models in this advanced course, with a focus on their reliability and trustworthiness. Students will engage with recent developments in the literature, focusing on vision-language models, and will also explore large language models and vision-audio models. Depending on time, the course may extend into the application of these models in generating visual, audio, and video content. The course will be structured with both readings of critical papers and expert-led presentations. Throughout the semester, each student will give one or two lectures. They must turn in a draft of their slide deck at least 24 hours in advance. Students will also have the chance to embark on a research project that will result in a 5-page paper written in the NeurIPS style.

Papers List: Google Doc
Schedule: TBD

Grading

The seminar is graded on a Satisfactory/Unsatisfactory (S/U) basis. A score of 75% or higher is considered Satisfactory, while a score of less than 75% is considered Unsatisfactory. The grading breakdown is as follows: 40% of the grade is based on presentations, and 60% is based on the project.

Project

For the project, students should form groups of 1 or 2 and aim to make a small but meaningful contribution to machine learning research. To generate ideas, they should explore recent conferences such as NeurIPS, ICML, CVPR, and ECCV/ICCV, and follow at least five recent papers within a specific thread. The final deliverable is a project report in the style of a NeurIPS paper, consisting of 5 pages including an abstract, body, and references. Students can download the LaTeX style file for the report here.