Eliciting and learning with soft labels from every annotator

KM Collins, U Bhatt, A Weller - Proceedings of the AAAI conference on …, 2022 - ojs.aaai.org
The labels used to train machine learning (ML) models are of paramount importance.
Typically for ML classification tasks, datasets contain hard labels, yet learning using soft …

Conformal prediction with large language models for multi-choice question answering

B Kumar, C Lu, G Gupta, A Palepu, D Bellamy… - arXiv preprint arXiv …, 2023 - arxiv.org
As large language models continue to be widely developed, robust uncertainty
quantification techniques will become crucial for their safe deployment in high-stakes …

Federated conformal predictors for distributed uncertainty quantification

C Lu, Y Yu, SP Karimireddy… - … on Machine Learning, 2023 - proceedings.mlr.press
Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty
quantification in machine learning since it can be easily applied as a post-processing step to …

Improving expert predictions with conformal prediction

E Straitouri, L Wang, N Okati… - … on Machine Learning, 2023 - proceedings.mlr.press
Automated decision support systems promise to help human experts solve multiclass
classification tasks more efficiently and accurately. However, existing systems typically …

Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles

R Verma, D Barrejón… - … Conference on Artificial …, 2023 - proceedings.mlr.press
We study the statistical properties of learning to defer (L2D) to multiple experts. In particular,
we address the open problems of deriving a consistent surrogate loss, confidence …

Length optimization in conformal prediction

S Kiyani, G Pappas, H Hassani - arXiv preprint arXiv:2406.18814, 2024 - arxiv.org
Conditional validity and length efficiency are two crucial aspects of conformal prediction
(CP). Achieving conditional validity ensures accurate uncertainty quantification for data …

Evaluating the utility of conformal prediction sets for ai-advised image labeling

D Zhang, A Chatzimparmpas, N Kamali… - Proceedings of the CHI …, 2024 - dl.acm.org
As deep neural networks are more commonly deployed in high-stakes domains, their black-
box nature makes uncertainty quantification challenging. We investigate the effects of …

Conformal prediction sets improve human decision making

JC Cresswell, Y Sui, B Kumar, N Vouitsis - arXiv preprint arXiv:2401.13744, 2024 - arxiv.org
In response to everyday queries, humans explicitly signal uncertainty and offer alternative
answers when they are unsure. Machine learning models that output calibrated prediction …

Designing decision support systems using counterfactual prediction sets

E Straitouri, MG Rodriguez - arXiv preprint arXiv:2306.03928, 2023 - arxiv.org
Decision support systems for classification tasks are predominantly designed to predict the
value of the ground truth labels. However, since their predictions are not perfect, these …

Learning personalized decision support policies

U Bhatt, V Chen, KM Collins, P Kamalaruban… - arXiv preprint arXiv …, 2023 - arxiv.org
Individual human decision-makers may benefit from different forms of support to improve
decision outcomes. However, a key question is which form of support will lead to accurate …