Dissociating language and thought in large language models
Large language models (LLMs) have come closest among all models to date to mastering
human language, yet opinions about their linguistic and cognitive capabilities remain split …
human language, yet opinions about their linguistic and cognitive capabilities remain split …
Embers of autoregression: Understanding large language models through the problem they are trained to solve
The widespread adoption of large language models (LLMs) makes it important to recognize
their strengths and limitations. We argue that in order to develop a holistic understanding of …
their strengths and limitations. We argue that in order to develop a holistic understanding of …
Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement
The ability to derive underlying principles from a handful of observations and then
generalize to novel situations--known as inductive reasoning--is central to human …
generalize to novel situations--known as inductive reasoning--is central to human …
LexSym: Compositionality as lexical symmetry
In tasks like semantic parsing, instruction following, and question answering, standard deep
networks fail to generalize compositionally from small datasets. Many existing approaches …
networks fail to generalize compositionally from small datasets. Many existing approaches …
How Do In-Context Examples Affect Compositional Generalization?
Compositional generalization--understanding unseen combinations of seen primitives--is an
essential reasoning capability in human intelligence. The AI community mainly studies this …
essential reasoning capability in human intelligence. The AI community mainly studies this …
Recogs: How incidental details of a logical form overshadow an evaluation of semantic interpretation
Compositional generalization benchmarks for semantic parsing seek to assess whether
models can accurately compute meanings for novel sentences, but operationalize this in …
models can accurately compute meanings for novel sentences, but operationalize this in …
Break it down: Evidence for structural compositionality in neural networks
Though modern neural networks have achieved impressive performance in both vision and
language tasks, we know little about the functions that they implement. One possibility is that …
language tasks, we know little about the functions that they implement. One possibility is that …
Language models as models of language
R Millière - arXiv preprint arXiv:2408.07144, 2024 - arxiv.org
This chapter critically examines the potential contributions of modern language models to
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …
COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models
A characteristic feature of human semantic cognition is its ability to not only store and
retrieve the properties of concepts observed through experience, but to also facilitate the …
retrieve the properties of concepts observed through experience, but to also facilitate the …
Language model acceptability judgements are not always robust to context
Targeted syntactic evaluations of language models ask whether models show stable
preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most …
preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most …