State-of-the-art generalisation research in NLP: a taxonomy and review
The ability to generalise well is one of the primary desiderata of natural language
processing (NLP). Yet, what'good generalisation'entails and how it should be evaluated is …
processing (NLP). Yet, what'good generalisation'entails and how it should be evaluated is …
[PDF][PDF] Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora
Children can acquire language from less than 100 million words of input. Large language
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
A theory of emergent in-context learning as implicit structure induction
Scaling large language models (LLMs) leads to an emergent capacity to learn in-context
from example demonstrations. Despite progress, theoretical understanding of this …
from example demonstrations. Despite progress, theoretical understanding of this …
Unit testing for concepts in neural networks
C Lovering, E Pavlick - Transactions of the Association for …, 2022 - direct.mit.edu
Many complex problems are naturally understood in terms of symbolic concepts. For
example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non …
example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non …
How abstract is linguistic generalization in large language models? Experiments with argument structure
Abstract Language models are typically evaluated on their success at predicting the
distribution of specific words in specific contexts. Yet linguistic knowledge also encodes …
distribution of specific words in specific contexts. Yet linguistic knowledge also encodes …
Grokking of hierarchical structure in vanilla transformers
For humans, language production and comprehension is sensitive to the hierarchical
structure of sentences. In natural language processing, past work has questioned how …
structure of sentences. In natural language processing, past work has questioned how …
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech
When acquiring syntax, children consistently choose hierarchical rules over competing non-
hierarchical possibilities. Is this preference due to a learning bias for hierarchical structure …
hierarchical possibilities. Is this preference due to a learning bias for hierarchical structure …
Language model acceptability judgements are not always robust to context
Targeted syntactic evaluations of language models ask whether models show stable
preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most …
preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most …
The Impact of Depth on Compositional Generalization in Transformer Language Models
To process novel sentences, language models (LMs) must generalize compositionally—
combine familiar elements in new ways. What aspects of a model's structure promote …
combine familiar elements in new ways. What aspects of a model's structure promote …
How to plant trees in language models: Data and architectural effects on the emergence of syntactic inductive biases
Accurate syntactic representations are essential for robust generalization in natural
language. Recent work has found that pre-training can teach language models to rely on …
language. Recent work has found that pre-training can teach language models to rely on …