Taxonomy of machine learning safety: A survey and primer

S Mohseni, H Wang, C Xiao, Z Yu, Z Wang… - ACM Computing …, 2022 - dl.acm.org
The open-world deployment of Machine Learning (ML) algorithms in safety-critical
applications such as autonomous vehicles needs to address a variety of ML vulnerabilities …

Agreement-on-the-line: Predicting the performance of neural networks under distribution shift

C Baek, Y Jiang, A Raghunathan… - Advances in Neural …, 2022 - proceedings.neurips.cc
Recently, Miller et al. showed that a model's in-distribution (ID) accuracy has a strong linear
correlation with its out-of-distribution (OOD) accuracy, on several OOD benchmarks, a …

Rankfeat: Rank-1 feature removal for out-of-distribution detection

Y Song, N Sebe, W Wang - Advances in Neural Information …, 2022 - proceedings.neurips.cc
The task of out-of-distribution (OOD) detection is crucial for deploying machine learning
models in real-world settings. In this paper, we observe that the singular value distributions …

Domain adaptation under open set label shift

S Garg, S Balakrishnan… - Advances in Neural …, 2022 - proceedings.neurips.cc
We introduce the problem of domain adaptation under Open Set Label Shift (OSLS), where
the label distribution can change arbitrarily and a new class may arrive during deployment …

Probvlm: Probabilistic adapter for frozen vison-language models

U Upadhyay, S Karthik, M Mancini… - Proceedings of the …, 2023 - openaccess.thecvf.com
Large-scale vision-language models (VLMs) like CLIP successfully find correspondences
between images and text. Through the standard deterministic mapping process, an image or …

Can you rely on your model evaluation? improving model evaluation with synthetic test data

B van Breugel, N Seedat, F Imrie… - Advances in Neural …, 2024 - proceedings.neurips.cc
Evaluating the performance of machine learning models on diverse and underrepresented
subgroups is essential for ensuring fairness and reliability in real-world applications …

Complementary benefits of contrastive learning and self-training under distribution shift

S Garg, A Setlur, Z Lipton… - Advances in …, 2024 - proceedings.neurips.cc
Self-training and contrastive learning have emerged as leading techniques for incorporating
unlabeled data, both under distribution shift (unsupervised domain adaptation) and when it …

Gnnevaluator: Evaluating gnn performance on unseen graphs without labels

X Zheng, M Zhang, C Chen, S Molaei… - Advances in Neural …, 2024 - proceedings.neurips.cc
Evaluating the performance of graph neural networks (GNNs) is an essential task for
practical GNN model deployment and serving, as deployed GNNs face significant …

A survey on evaluation of out-of-distribution generalization

H Yu, J Liu, X Zhang, J Wu, P Cui - arXiv preprint arXiv:2403.01874, 2024 - arxiv.org
Machine learning models, while progressively advanced, rely heavily on the IID assumption,
which is often unfulfilled in practice due to inevitable distribution shifts. This renders them …

Rlsbench: Domain adaptation under relaxed label shift

S Garg, N Erickson, J Sharpnack… - International …, 2023 - proceedings.mlr.press
Despite the emergence of principled methods for domain adaptation under label shift, their
sensitivity to shifts in class conditional distributions is precariously under explored …