Dissociating language and thought in large language models

K Mahowald, AA Ivanova, IA Blank, N Kanwisher… - Trends in Cognitive …, 2024 - cell.com
Large language models (LLMs) have come closest among all models to date to mastering
human language, yet opinions about their linguistic and cognitive capabilities remain split …

Embers of autoregression: Understanding large language models through the problem they are trained to solve

RT McCoy, S Yao, D Friedman, M Hardy… - arXiv preprint arXiv …, 2023 - arxiv.org
The widespread adoption of large language models (LLMs) makes it important to recognize
their strengths and limitations. We argue that in order to develop a holistic understanding of …

Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement

L Qiu, L Jiang, X Lu, M Sclar, V Pyatkin… - arXiv preprint arXiv …, 2023 - arxiv.org
The ability to derive underlying principles from a handful of observations and then
generalize to novel situations--known as inductive reasoning--is central to human …

LexSym: Compositionality as lexical symmetry

E Akyürek, J Andreas - Proceedings of the 61st Annual Meeting of …, 2023 - aclanthology.org
In tasks like semantic parsing, instruction following, and question answering, standard deep
networks fail to generalize compositionally from small datasets. Many existing approaches …

How Do In-Context Examples Affect Compositional Generalization?

S An, Z Lin, Q Fu, B Chen, N Zheng, JG Lou… - arXiv preprint arXiv …, 2023 - arxiv.org
Compositional generalization--understanding unseen combinations of seen primitives--is an
essential reasoning capability in human intelligence. The AI community mainly studies this …

Recogs: How incidental details of a logical form overshadow an evaluation of semantic interpretation

Z Wu, CD Manning, C Potts - Transactions of the Association for …, 2023 - direct.mit.edu
Compositional generalization benchmarks for semantic parsing seek to assess whether
models can accurately compute meanings for novel sentences, but operationalize this in …

Break it down: Evidence for structural compositionality in neural networks

M Lepori, T Serre, E Pavlick - Advances in Neural …, 2023 - proceedings.neurips.cc
Though modern neural networks have achieved impressive performance in both vision and
language tasks, we know little about the functions that they implement. One possibility is that …

Language models as models of language

R Millière - arXiv preprint arXiv:2408.07144, 2024 - arxiv.org
This chapter critically examines the potential contributions of modern language models to
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …

COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models

K Misra, JT Rayz, A Ettinger - arXiv preprint arXiv:2210.01963, 2022 - arxiv.org
A characteristic feature of human semantic cognition is its ability to not only store and
retrieve the properties of concepts observed through experience, but to also facilitate the …

Language model acceptability judgements are not always robust to context

K Sinha, J Gauthier, A Mueller, K Misra… - arXiv preprint arXiv …, 2022 - arxiv.org
Targeted syntactic evaluations of language models ask whether models show stable
preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most …