Online crowdsourcing provides a scalable and inexpensive means to collect knowledge (eg
labels) about various types of data items (eg text, audio, video). However, it is also known to …

Systematic literature reviews (SLRs) are one of the most common and useful form of
scientific research and publication. Tens of thousands of SLRs are published each year, and …

In this tutorial, we present a portion of unique industry experience in efficient data labeling
via crowdsourcing shared by both leading researchers and engineers from Yandex. We will …

Quality control plays a critical role in crowdsourcing. The state-of-the-art work is not suitable
for crowdsourcing applications that require extensive validation of the tasks quality, since it …

Modern machine learning algorithms need large datasets to be trained. Crowdsourcing has
become a popular approach to label large datasets in a shorter time as well as at a lower …

In this tutorial, we present a portion of unique industry experience in efficient natural
language data annotation via crowdsourcing shared by both leading researchers and …

In this tutorial, we present a portion of unique industry experience in efficient data labelling
via crowdsourcing shared by both leading researchers and engineers from Yandex. We will …

Many annotation systems provide to add an unsure option in the labels, because the
annotators have different expertise, and they may not have enough confidence to choose a …

We study the problem of ranking from crowdsourced pairwise comparisons. Answers to
pairwise tasks are known to be affected by the position of items on the screen, however …

Crowdsourcing has proved its ability to address large scale data collection tasks at a low
cost and in a short time. However, due to the dependence on unknown workers, the quality …