The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning

J Hullman, S Kapoor, P Nanayakkara… - Proceedings of the …, 2022 - dl.acm.org
Arguments that machine learning (ML) is facing a reproducibility and replication crisis
suggest that some published claims in research cannot be taken at face value. Concerns …

Reverse‐Bayes methods for evidence assessment and research synthesis

L Held, R Matthews, M Ott… - Research Synthesis …, 2022 - Wiley Online Library
It is now widely accepted that the standard inferential toolkit used by the scientific research
community—null‐hypothesis significance testing (NHST)—is not fit for purpose. Yet despite …

Replicability across multiple studies

M Bogomolov, R Heller - Statistical Science, 2023 - projecteuclid.org
Replicability Across Multiple Studies Page 1 Statistical Science 2023, Vol. 38, No. 4, 602–620
https://doi.org/10.1214/23-STS892 © Institute of Mathematical Statistics, 2023 Replicability …

Bayes factors for peri-null hypotheses

A Ly, EJ Wagenmakers - Test, 2022 - Springer
A perennial objection against Bayes factor point-null hypothesis tests is that the point-null
hypothesis is known to be false from the outset. We examine the consequences of …

Power priors for replication studies

S Pawel, F Aust, L Held, EJ Wagenmakers - Test, 2024 - Springer
The ongoing replication crisis in science has increased interest in the methodology of
replication studies. We propose a novel Bayesian analysis approach using power priors …

Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

U Schimmack, F Bartoš - Plos one, 2023 - journals.plos.org
The influential claim that most published results are false raised concerns about the
trustworthiness and integrity of science. Since then, there have been numerous attempts to …

Bayesian approaches to designing replication studies.

S Pawel, G Consonni, L Held - Psychological Methods, 2023 - psycnet.apa.org
Replication studies are essential for assessing the credibility of claims from original studies.
A critical aspect of designing replication studies is determining their sample size; a too-small …

Prediction scoring of data-driven discoveries for reproducible research

AL Smith, T Zheng, A Gelman - Statistics and Computing, 2023 - Springer
Predictive modeling uncovers knowledge and insights regarding a hypothesized data
generating mechanism (DGM). Results from different studies on a complex DGM, derived …

Assessing replicability with the sceptical pp‐value: Type‐I error control and sample size planning

C Micheloud, F Balabdaoui, L Held - Statistica Neerlandica, 2023 - Wiley Online Library
We study a statistical framework for replicability based on a recently proposed quantitative
measure of replication success, the sceptical pp‐value. A recalibration is proposed to obtain …

Closed-Form Power and Sample Size Calculations for Bayes Factors

S Pawel, L Held - arXiv preprint arXiv:2406.19940, 2024 - arxiv.org
Determining an appropriate sample size is a critical element of study design, and the
method used to determine it should be consistent with the planned analysis. When the …