We investigate a surprising limitation of LLMs: their inability to consistently generate text in a
user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate …

Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-
resource language tasks, yet their performance in low-resource languages is hindered by …

Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their
increasing global use. In this work, we explore zero-shot cross-lingual generalization of …

Preference optimization techniques have become a standard final stage for training state-of-
art large language models (LLMs). However, despite widespread adoption, the vast majority …

Quantization techniques are widely used to improve inference speed and deployment of
large language models. While a wide body of work examines the impact of quantized LLMs …

Words have been represented in a high-dimensional vector space that encodes their
semantic similarities, enabling downstream applications such as retrieving synonyms …

Medical knowledge is context-dependent and requires consistent reasoning across various
natural language expressions of semantically equivalent phrases. This is particularly crucial …

WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language
models (LLMs). It accommodates a diverse range of models, including both open-weight …

Generalization and robustness to input variation are core desiderata of machine learning
research. Language varies along several axes, most importantly, language instance (eg …

Large language models (LLMs) often lack culture-specific knowledge of daily life, especially
across diverse regions and non-English languages. Existing benchmarks for evaluating …