Eliciting and learning with soft labels from every annotator
The labels used to train machine learning (ML) models are of paramount importance.
Typically for ML classification tasks, datasets contain hard labels, yet learning using soft …
Typically for ML classification tasks, datasets contain hard labels, yet learning using soft …
Conformal prediction with large language models for multi-choice question answering
As large language models continue to be widely developed, robust uncertainty
quantification techniques will become crucial for their safe deployment in high-stakes …
quantification techniques will become crucial for their safe deployment in high-stakes …
Federated conformal predictors for distributed uncertainty quantification
Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty
quantification in machine learning since it can be easily applied as a post-processing step to …
quantification in machine learning since it can be easily applied as a post-processing step to …
Improving expert predictions with conformal prediction
Automated decision support systems promise to help human experts solve multiclass
classification tasks more efficiently and accurately. However, existing systems typically …
classification tasks more efficiently and accurately. However, existing systems typically …
Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles
R Verma, D Barrejón… - … Conference on Artificial …, 2023 - proceedings.mlr.press
We study the statistical properties of learning to defer (L2D) to multiple experts. In particular,
we address the open problems of deriving a consistent surrogate loss, confidence …
we address the open problems of deriving a consistent surrogate loss, confidence …
Length optimization in conformal prediction
Conditional validity and length efficiency are two crucial aspects of conformal prediction
(CP). Achieving conditional validity ensures accurate uncertainty quantification for data …
(CP). Achieving conditional validity ensures accurate uncertainty quantification for data …
Evaluating the utility of conformal prediction sets for ai-advised image labeling
As deep neural networks are more commonly deployed in high-stakes domains, their black-
box nature makes uncertainty quantification challenging. We investigate the effects of …
box nature makes uncertainty quantification challenging. We investigate the effects of …
Conformal prediction sets improve human decision making
In response to everyday queries, humans explicitly signal uncertainty and offer alternative
answers when they are unsure. Machine learning models that output calibrated prediction …
answers when they are unsure. Machine learning models that output calibrated prediction …
Designing decision support systems using counterfactual prediction sets
E Straitouri, MG Rodriguez - arXiv preprint arXiv:2306.03928, 2023 - arxiv.org
Decision support systems for classification tasks are predominantly designed to predict the
value of the ground truth labels. However, since their predictions are not perfect, these …
value of the ground truth labels. However, since their predictions are not perfect, these …
Learning personalized decision support policies
Individual human decision-makers may benefit from different forms of support to improve
decision outcomes. However, a key question is which form of support will lead to accurate …
decision outcomes. However, a key question is which form of support will lead to accurate …