Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model
size as a function of the compute budget, but these laws yield substantially different …

It is an open challenge to obtain high quality training data, especially captions, for text-to-
audio models. Although prior methods have leveraged\textit {text-only language models} to …

Multimodal foundation models serve numerous applications at the intersection of vision and
language. Still, despite being pretrained on extensive data, they become outdated over time …

Low precision training and inference affect both the quality and cost of language models, but
current scaling laws do not account for this. In this work, we devise" precision-aware" scaling …

Encoder-only transformer models such as BERT offer a great performance-size tradeoff for
retrieval and classification tasks with respect to larger decoder-only models. Despite being …

Large Language Models have driven significant AI advancements, yet their training is
resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws …

Finding the optimal learning rate for language model pretraining is a challenging task. This
is not only because there is a complicated correlation between learning rate, batch size …

The performance of embodied agents has been shown to improve by increasing model
parameters, dataset size, and compute. This has been demonstrated in domains from …

Pretraining data selection has the potential to improve language model pretraining efficiency
by utilizing higher-quality data from massive web data corpora. Current data selection …

This paper investigates the under-explored area of low-rank weight training for large-scale
Conformer-based speech recognition models from scratch. Our study demonstrates the …