Word class representations spontaneously emerge in a deep neural network trained on next word prediction

K Surendra, A Schilling, P Stoewer… - 2023 International …, 2023 - ieeexplore.ieee.org
How do humans learn language, and can the first language be learned at all? These
fundamental questions are still hotly debated. In contemporary linguistics, there are two …

Artificial neural network language models align neurally and behaviorally with humans even after a developmentally realistic amount of training

EA Hosseini, M Schrimpf, Y Zhang, S Bowman… - BioRxiv, 2022 - biorxiv.org
Artificial neural networks have emerged as computationally plausible models of human
language processing. A major criticism of these models is that the amount of training data …

Tying word vectors and word classifiers: A loss framework for language modeling

H Inan, K Khosravi, R Socher - arXiv preprint arXiv:1611.01462, 2016 - arxiv.org
Recurrent neural networks have been very successful at predicting sequences of words in
tasks such as language modeling. However, all such models are based on the conventional …

The human unlikeness of neural language models in next-word prediction

CL Jacobs, AD McCarthy - Proceedings of the Fourth Widening …, 2020 - aclanthology.org
The training objective of unidirectional language models (LMs) is similar to a
psycholinguistic benchmark known as the cloze task, which measures next-word …

Predict the Next Word

E Ilia, W Aziz - arXiv preprint arXiv:2402.17527, 2024 - arxiv.org
Language models (LMs) are statistical models trained to assign probability to human-
generated text. As such, it is reasonable to question whether they approximate linguistic …

Frustratingly short attention spans in neural language modeling

M Daniluk, T Rocktäschel, J Welbl, S Riedel - arXiv preprint arXiv …, 2017 - arxiv.org
Neural language models predict the next token using a latent representation of the
immediate token history. Recently, various methods for augmenting neural language models …

A sentence is worth a thousand pictures: Can large language models understand human language?

G Marcus, E Leivada, E Murphy - arXiv preprint arXiv:2308.00109, 2023 - arxiv.org
Artificial Intelligence applications show great potential for language-related tasks that rely on
next-word prediction. The current generation of large language models have been linked to …

Language acquisition: do children and language models follow similar learning stages?

L Evanson, Y Lakretz, JR King - arXiv preprint arXiv:2306.03586, 2023 - arxiv.org
During language acquisition, children follow a typical sequence of learning stages, whereby
they first learn to categorize phonemes before they develop their lexicon and eventually …

Analysis of Argument Structure Constructions in a Deep Recurrent Language Model

P Ramezani, A Schilling, P Krauss - arXiv preprint arXiv:2408.03062, 2024 - arxiv.org
Understanding how language and linguistic constructions are processed in the brain is a
fundamental question in cognitive computational neuroscience. In this study, we explore the …

[PDF][PDF] Predicting Age of Acquisition in Early Word Learning Using Recurrent Neural Networks.

E Portelance, J Degen, MC Frank - CogSci, 2020 - cognitivesciencesociety.org
Vocabulary growth and syntactic development are known to be highly correlated in early
child language. What determines when words are acquired and how can this help us …