Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization

arXiv2305.11965MaRDI QIDQ6437274

Author name not available (Why is that?)

Publication date: 19 May 2023

Abstract: In this paper, we aim to optimize a contrastive loss with individualized temperatures in a principled and systematic manner for self-supervised learning. The common practice of using a global temperature parameter $au$ ignores the fact that ``not all semantics are created equal", meaning that different anchor data may have different numbers of samples with similar semantics, especially when data exhibits long-tails. First, we propose a new robust contrastive loss inspired by distributionally robust optimization (DRO), providing us an intuition about the effect of $au$ and a mechanism for automatic temperature individualization. Then, we propose an efficient stochastic algorithm for optimizing the robust contrastive loss with a provable convergence guarantee without using large mini-batch sizes. Theoretical and experimental results show that our algorithm automatically learns a suitable $au$ for each sample. Specifically, samples with frequent semantics use large temperatures to keep local semantic structures, while samples with rare semantics use small temperatures to induce more separable features. Our method not only outperforms prior strong baselines (e.g., SimCLR, CLIP) on unimodal and bimodal datasets with larger improvements on imbalanced data but also is less sensitive to hyper-parameters. To our best knowledge, this is the first methodical approach to optimizing a contrastive loss with individualized temperatures.

Has companion code repository: https://github.com/zhqiu/contrastive-learning-isogclr

This page was built for publication: Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6437274)