Human variation in labeling is often considered noise. Annotation projects for machine
learning (ML) aim at minimizing human label variation, with the assumption to maximize …

Grounded text generation systems often generate text that contains factual inconsistencies,
hindering their real-world applicability. Automatic factual consistency evaluation may help …

Large language models (LLMs) are increasingly becoming all-powerful and pervasive via
deployment in sociotechnical systems. Yet these language models, be it for classification or …

Computer vision models have known performance disparities across attributes such as
gender and skin tone. This means during tasks such as classification and detection, model …

This paper presents a community-centered study of cultural limitations of text-to-image (T2I)
models in the South Asian context. We theorize these failures using scholarship on …

Technology companies continue to invest in efforts to incorporate responsibility in their
Artificial Intelligence (AI) advancements, while efforts to audit and regulate AI systems …

Abstract Machine learning approaches often require training and evaluation datasets with a
clear separation between positive and negative examples. This requirement overly …

Biomedical image analysis algorithm validation depends on high-quality annotation of
reference datasets, for which labelling instructions are key. Despite their importance, their …

A growing literature on human-AI decision-making investigates strategies for combining
human judgment with statistical models to improve decision-making. Research in this area …

To investigate the well-observed racial disparities in computer vision systems that analyze
images of humans, researchers have turned to skin tone as a more objective annotation …