A survey on stability of learning with limited labelled data and its sensitivity to the effects of randomness
Learning with limited labelled data, such as prompting, in-context learning, fine-tuning, meta-
learning, or few-shot learning, aims to effectively train a model using only a small amount of …
learning, or few-shot learning, aims to effectively train a model using only a small amount of …
A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations
Abstract Large Language Models (LLMs) have recently gained significant attention due to
their remarkable capabilities in performing diverse tasks across various domains. However …
their remarkable capabilities in performing diverse tasks across various domains. However …
Refusal in language models is mediated by a single direction
Conversational large language models are fine-tuned for both instruction-following and
safety, resulting in models that obey benign requests but refuse harmful ones. While this …
safety, resulting in models that obey benign requests but refuse harmful ones. While this …
Open problems in technical ai governance
AI progress is creating a growing range of risks and opportunities, but it is often unclear how
they should be navigated. In many cases, the barriers and uncertainties faced are at least …
they should be navigated. In many cases, the barriers and uncertainties faced are at least …
[PDF][PDF] CALAMITA: Challenge the Abilities of LAnguage Models in ITAlian
The rapid development of Large Language Models (LLMs) has called for robust benchmarks
to assess their abilities, track progress, and compare iterations. While existing benchmarks …
to assess their abilities, track progress, and compare iterations. While existing benchmarks …
Lab-bench: Measuring capabilities of language models for biology research
There is widespread optimism that frontier Large Language Models (LLMs) and LLM-
augmented systems have the potential to rapidly accelerate scientific discovery across …
augmented systems have the potential to rapidly accelerate scientific discovery across …
Composable interventions for language models
Test-time interventions for language models can enhance factual accuracy, mitigate harmful
outputs, and improve model efficiency without costly retraining. But despite a flood of new …
outputs, and improve model efficiency without costly retraining. But despite a flood of new …
The responsible foundation model development cheatsheet: A review of tools & resources
Foundation model development attracts a rapidly expanding body of contributors, scientists,
and applications. To help shape responsible development practices, we introduce the …
and applications. To help shape responsible development practices, we introduce the …
LLM Stability: A detailed analysis with some surprises
LLM (large language model) practitioners commonly notice that outputs can vary for the
same inputs, but we have been unable to find work that evaluates LLM stability as the main …
same inputs, but we have been unable to find work that evaluates LLM stability as the main …
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
AI models are increasingly prevalent in high-stakes environments, necessitating thorough
assessment of their capabilities and risks. Benchmarks are popular for measuring these …
assessment of their capabilities and risks. Benchmarks are popular for measuring these …