Bayesian distillation of deep learning models
type: Item
instance of: scholarly article
title: Bayesian distillation of deep learning models
zbMATH Open document ID: 1491.68180
DOI: 10.1134/S0005117921110023
author: A. V. Grabovoy
author: Vadim Strijov
published in: Automation and Remote Control
publication date: 2022-01-21T00:00:00Z
review text: The authors present a Bayesian approach to teacher-student networks' knowledge distillation. Knowledge distillation was first proposed by \textit{G. Hinton} et al. in their paper [``Distilling the knowledge in a neural network'', Preprint, \url{arXiv:1503.02531}]. They proposed to train a large network with ground truth labels as the teacher network, then train a smaller model on the outputs of the teacher network as ``soft targets''. This work extends the prior framework of teacher-student networks. The authors argue that the parameters of the student network can be initialized from the teacher network. The teacher network is usually larger than the student network. To meaningfully initialize the student network, the authors propose to prune the teacher network so that it has the same architecture as the student network. With the assumption that the posterior of the teacher network follows a Gaussian distribution, the authors prove that the pruned teacher network also follows a Gaussian distribution.
reviewed by: Hongshan Li
Mathematics Subject Classification ID: 68T07
Mathematics Subject Classification ID: 62F15
zbMATH DE Number: 7461105
zbMATH Keywords: deep learning
zbMATH Keywords: Bayesian methods
zbMATH Keywords: knowledge distillation
zbMATH Keywords: model selection
zbMATH Keywords: Bayesian inference
describes a project that uses: ImageNet
describes a project that uses: BERT
describes a project that uses: AlexNet
describes a project that uses: Fashion-MNIST
describes a project that uses: MNIST
describes a project that uses: Tensor2Tensor
describes a project that uses: mT5
describes a project that uses: GPT-3
MaRDI profile type: Publication
cites work: Q5744830
full work available at URL: s0005117921110023
OpenAlex ID: W4200392901
Recommended article: Deep learning: a Bayesian perspective
Recommended article: Bayesian learning for neural networks
Recommended article: Bayesian Deep Net GLM and GLMM
Recommended article: Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network
Recommended article: Priors in Bayesian Deep Learning: A Review
Recommended article: Bayesian learning for recurrent neural networks
Recommended article: Deep Variational Inference
Recommended article: Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification
Recommended article: Bayesian inference in neural networks
description: scientific article; zbMATH DE number 7461105