Human variation in labeling is often considered noise. Annotation projects for machine
learning (ML) aim at minimizing human label variation, with the assumption to maximize …

Abstract Homophobia and Transphobia Detection is the task of identifying homophobia,
transphobia, and non-anti-LGBT+ content from the given corpus. Homophobia and …

Machine learning models are commonly used to detect toxicity in online conversations.
These models are trained on datasets annotated by human raters. We explore how raters' …

This paper makes a case for addressing humour as an online safety issue so that social
media platforms can include it in their risk assessments and harm mitigation strategies. We …

Annotators' sociodemographic backgrounds (ie, the individual compositions of their gender,
age, educational background, etc.) have a strong impact on their decisions when working on …

Abstract We introduce the Measuring Hate Speech corpus, a dataset created to measure
hate speech while adjusting for annotators' perspectives. It consists of 50,070 social media …

Many NLP tasks exhibit human label variation, where different annotators give different
labels to the same texts. This variation is known to depend, at least in part, on the …

Though majority vote among annotators is typically used for ground truth labels in natural
language processing, annotator disagreement in tasks such as hate speech detection may …

Recent studies have shown that for subjective annotation tasks, the demographics, lived
experiences, and identity of annotators can have a large impact on how items are labeled …

This paper provides a comprehensive summary of the “Homophobia and Transphobia
Detection in Social Media Comments” shared task, which was held at the LT-EDI@ EACL …