Invariance, Encodings, and Generalization: Learning Identity Effects With Neural Networks
From MaRDI portal
Publication:5095898
DOI10.1162/NECO_A_01510zbMATH Open1494.92004arXiv2101.08386OpenAlexW3123042577WikidataQ114926512 ScholiaQ114926512MaRDI QIDQ5095898
Author name not available (Why is that?)
Publication date: 11 August 2022
Published in: Neural Computation (Search for Journal in Brave)
Abstract: Often in language and other areas of cognition, whether two components of an object are identical or not determines if it is well formed. We call such constraints identity effects. When developing a system to learn well-formedness from examples, it is easy enough to build in an identify effect. But can identity effects be learned from the data without explicit guidance? We provide a framework in which we can rigorously prove that algorithms satisfying simple criteria cannot make the correct inference. We then show that a broad class of learning algorithms including deep feedforward neural networks trained via gradient-based algorithms (such as stochastic gradient descent or the Adam method) satisfy our criteria, dependent on the encoding of inputs. In some broader circumstances we are able to provide adversarial examples that the network necessarily classifies incorrectly. Finally, we demonstrate our theory with computational experiments in which we explore the effect of different input encodings on the ability of algorithms to generalize to novel inputs. This allows us to show similar effects to those predicted by theory for more realistic methods that violate some of the conditions of our theoretical results.
Full work available at URL: https://arxiv.org/abs/2101.08386
Recommendations
- Title not available (Why is that?) 👍 👎
- Title not available (Why is that?) 👍 👎
- Neural learning methods yielding functional invariance 👍 👎
- Representation theory and invariant neural networks 👍 👎
- Neurally plausible mechanisms for learning selective and invariant representations 👍 👎
- An embedding of ReLU networks and an analysis of their identifiability 👍 👎
- Learning Invariant Features in Modulatory Networks through Conflict and Ambiguity 👍 👎
- Invariance Properties of Recurrent Neural Networks 👍 👎
- On invariance and selectivity in representation learning 👍 👎
This page was built for publication: Invariance, Encodings, and Generalization: Learning Identity Effects With Neural Networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5095898)
