An enriched category theory of language: from syntax to semantics (Q2153136)

Large language models (LLMs) have recently attained new levels of sophistication by effectively learning a probability distribution on possible continuations of a given text [\textit{A. Vaswani} et al., ``Attention is all you need'', Preprint, \url{arXiv:1706.03762}, \url{https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf}, \url{https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf}; \textit{T. B. Brown}, ``Language models are few-shot learners'', Preprint, \url{arXiv:2005.14165}], interactively inputting prefix text and then sampling repeatedly from a text word distribution to generate original, high quality texts. This paper defines a syntax category, which is a category enriched over the unit interval \([0,1]\) modelling probability distributions on text continuations [\textit{T.-D. Bradley} and \textit{Y. Vlassopoulos}, Compositionality 3, No. 4, 21 p. (2021; Zbl 1489.91211)]. The semantic category is the enriched cateogry of \([0,1]\)-valued copresheaves on the syntax category, of which the syntax category is to be regarded as a subcategory via the Yoneda embedding. The \([0,1]\)-valued copresheaf represented by a text can be put down as the meaning of the text, as in dynamic semantics [\textit{R. Nouwen}, J. Philos. Log. 36, No. 2, 123--154 (2007; Zbl 1117.03335)]. There are categorical operations in the semantic category allowing of combining meanings that correspond to certain logical operations.