The llama 3 herd of models
Modern artificial intelligence (AI) systems are powered by foundation models. This paper
presents a new set of foundation models, called Llama 3. It is a herd of language models …
presents a new set of foundation models, called Llama 3. It is a herd of language models …
Llamafactory: Unified efficient fine-tuning of 100+ language models
Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks.
However, it requires non-trivial efforts to implement these methods on different models. We …
However, it requires non-trivial efforts to implement these methods on different models. We …
Benchmarking benchmark leakage in large language models
Amid the expanding use of pre-training data, the phenomenon of benchmark dataset
leakage has become increasingly prominent, exacerbated by opaque training processes …
leakage has become increasingly prominent, exacerbated by opaque training processes …
Advancing llm reasoning generalists with preference trees
We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning.
Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art …
Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art …
Common 7b language models already possess strong math capabilities
Mathematical capabilities were previously believed to emerge in common language models
only at a very large scale or require extensive math-related pre-training. This paper shows …
only at a very large scale or require extensive math-related pre-training. This paper shows …
Rho-1: Not all tokens are what you need
Previous language model pre-training methods have uniformly applied a next-token
prediction loss to all training tokens. Challenging this norm, we posit that" Not all tokens in a …
prediction loss to all training tokens. Challenging this norm, we posit that" Not all tokens in a …
Compression represents intelligence linearly
There is a belief that learning to compress well will lead to intelligence. Recently, language
modeling has been shown to be equivalent to compression, which offers a compelling …
modeling has been shown to be equivalent to compression, which offers a compelling …
Map-neo: Highly capable and transparent bilingual large language model series
Large Language Models (LLMs) have made great strides in recent years to achieve
unprecedented performance across different tasks. However, due to commercial interest, the …
unprecedented performance across different tasks. However, due to commercial interest, the …
Towards modular llms by building and reusing a library of loras
The growing number of parameter-efficient adaptations of a base large language model
(LLM) calls for studying whether we can reuse such trained adapters to improve …
(LLM) calls for studying whether we can reuse such trained adapters to improve …
Reverse training to nurse the reversal curse
Large language models (LLMs) have a surprising failure: when trained on" A has a feature
B", they do not generalize to" B is a feature of A", which is termed the Reversal Curse. Even …
B", they do not generalize to" B is a feature of A", which is termed the Reversal Curse. Even …