[Statlist] ETH/UZH Research Seminar on Statistics by Susan Wei, Monash University, Australia -- ETH Zurich, 29 April 2025

[Maurer Letizia]

We are glad to announce and invite you to the following talk in the ETH/UZH Research Seminar on Statistics:

"What's Degeneracy Got to Do with It? Understanding Deep Neural Networks through Singular Learning Theory"  
by Susan Wei, Monash University, Australia

Date and Time: Tuesday, 29 April 2025 at 13.15 h
Place: ETH Zurich, HG G 19.2

Abstract: Deep neural networks (DNNs) are singular statistical models whose loss landscapes exhibit complex degeneracies - features that defy explanation through classical regular statistical theory. Drawing on tools from singular learning theory, we introduce the Local Learning Coefficient (LLC), a quantity that rigorously quantifies the degree of degeneracy in DNNs. While the LLC is rooted in the singular framework, it recovers familiar notions of model complexity in regular or "minimally singular" regimes. We develop a scalable estimator for the LLC and apply it across a range of architectures, including deep linear networks with up to 100M parameters, ResNet image classifiers, and transformers. Empirical results demonstrate that the LLC sheds light on a range of deep learning phenomena, including in-context learning in transformers and the competition between energy and entropy during training dynamics, as well as how standard training heuristics influence effective model complexity. Ultimately, the LLC provides a practical instantiation of singular learning theory in modern deep learning, offering new perspectives on the interplay between overparameterization, generalization, and parsimony.

Seminar website, https://math.ethz.ch/sfs/news-and-events/research-seminar.html


Organisers:  A. Bandeira, P. Bühlmann, Y. Chen, R. Furrer, L. Held, T. Hothorn, D. Kozbur,  J. Peters,  M. Wolf, J. Ziegel