In supervised learning model development, domain experts are often used to provide the
class labels (annotations). Annotation inconsistencies commonly occur when even highly …

Despite sequences being core to NLP, scant work has considered how to handle noisy
sequence labels from multiple annotators for the same text. Given such annotations, we …

Because individual crowd workers often exhibit high variance in annotation accuracy, we
often ask multiple crowd workers to label each example to infer a single consensus label …

Although traditionally the primary information sources for cancer patients have been the
treating medical team, patients and their relatives increasingly turn to the Internet, though …

Modern search engines record user interactions and use them to improve search quality. In
particular, user click-through has been successfully used to improve clickthrough rate (CTR) …

Click-through information is considered as a valuable source of users' implicit relevance
feedback. As user behavior is usually influenced by a number of factors such as position …

Click-through information is considered as a valuable source of users' implicit relevance
feedback for commercial search engines. As existing studies have shown that the search …

By incorporating human workers into the query execution process crowd-enabled databases
facilitate intelligent, social capabilities like completing missing data at query time or …

Crowdsourcing [5] methods are quickly changing the landscape for the quantity, quality, and
type of labeled data available to supervised learning. While such data can now be obtained …

In this paper, we present the LiquidCrowd methodological approach based on consensus
and trustworthiness techniques for managing the execution of collective tasks. By collective …