The training of large language models (LLMs) is expensive. In this paper, we study data-
efficient approaches for pre-training LLMs, ie, techniques that aim to optimize the Pareto
frontier of model quality and training resource/data consumption. We seek to understand the
tradeoffs associated with data selection routines based on (i) expensive-to-compute data-
quality estimates, and (ii) maximization of coverage and diversity-based measures in the
feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities …