[Statlist] Speaker: Dr. Luigi Gresele (University of Copenhagen)

[Maurer Letizia]

Dear all,

You are cordially invited to the following in-person presentation, see below for details.
The talk will be of particular interest to people working on representation learning, identifiability, causality, or LLMs.
If you are interested in a 1:1 chat with Luigi, please send me a DM (vjulius at ethz.ch).

Best regards,
Julius


Title: Understanding Linear Properties and Similarity of Learned Representations via Identifiability Theory

Speaker: Dr. Luigi Gresele (University of Copenhagen)

Time & Date: 11:00-12:00, Thu 18 September, 2025

Location: HG G19.1 (Rämistrasse 101, 8092 Zürich)

Abstract: I will present two works that use identifiability theory to analyze representation learning, focusing on a model family that includes popular pre-training methods like autoregressive language models.
(i) The first work studies the ubiquity of linear properties observed in language models—e.g., the vector difference between "easy" and "easiest" being parallel to that between "lucky" and "luckiest." We ask whether a linear property found in one model must also appear in all models with the same next-token distribution. We prove an identifiability result for next-token predictors and show that, under suitable conditions, these linear properties appear in either all or none of them.
(ii) The second work asks when and why different models learn similar representations. We show that a small Kullback–Leibler divergence between model distributions does not guarantee representational similarity. We then introduce a new distributional distance and a similarity measure for which closeness does imply similarity—and find that wider networks learn distributions that are closer in this sense, leading to more similar representations.

References:
[1] https://arxiv.org/abs/2410.23501
[2] https://arxiv.org/abs/2506.03784

Bio: Luigi Gresele is a postdoc at the University of Copenhagen, with a fellowship from the Danish Data Science Academy (DDSA). His research focuses on machine learning and causality, and in particular on identifiability in representation learning and causal inference. Luigi earned his Ph.D. from the Max Planck Institute for Intelligent Systems and the University of Tübingen. During his Ph.D., he took part in an ELLIS exchange with the Parietal team at Inria-CEA and did an internship at Amazon Research. In 2024, he received an ELLIS PhD award for “outstanding dissertations in the field of modern AI”.

<[Invited Talk] L. Gresele (U Copenhagen) - -Understanding Linear Properties and Similarity of Learned Representations via Identifiability Theory- [In-person].ics>