A0528
Title: Self-supervised learning in the kernel regime
Authors: Maximilian Fleissner - Technical University of Munich (Germany) [presenting]
Abstract: In recent years, self-supervised learning (SSL) has emerged as a powerful paradigm, building the foundation of several modern machine learning models. At its core, SSL relies on the idea of using data augmentations to encode a notion of similarity in otherwise unlabeled samples. However, despite its rapidly growing popularity, the statistical understanding of what SSL learns is still limited. Arguably, one of the most promising avenues towards understanding the fundamental principles of SSL is by connecting it to kernel methods. In supervised learning, this correspondence is justified by virtue of the neural tangent kernel (NTK), which asserts that overparameterized neural networks follow training dynamics that resemble those of kernel machines. An extension of the NTK to a commonly used SSL algorithm is presented, namely, Barlow Twins. The NTK connection allows characterizing the patterns learned in SSL, as well as quantifying their generalization properties.
