The “Big Data” revolution is spawning systems designed to make decisions from data. In particular, deep learning methods have emerged as the state of the art method in many important breakthroughs [18, 20, 28]. This is due to the statistical flexibility and computational scalability of large and deep neural networks which allows them to harness the information of large and rich datasets. At the same time, elementary decision theory shows that the only admissible decision rules are Bayesian [5, 30]. Colloquially, this means that any decision rule which is not Bayesian can be strictly improved (or even exploited) by some Bayesian alternative [6]. The implication of these results is clear: combine deep learning with Bayesian inference for the best decisions from data.

There is a persistent history of research in Bayesian neural nets which never quite gained mainstream traction [19, 21]. The majority of deep learning research has evolved outside of Bayesian (or for that matter statistical) analysis [26, 18]. Recently, Bayesian deep learning has experienced a resurgence of interest [16, 1, 12]; one explanation for this revival is the rise of automated deep learning decision systems for which effective uncertainty estimates are essential [30]. In this paper we investigate several popular approaches for uncertainty estimation in neural networks. We find that several popular approximations to the uncertainty of a unknown neural net model are in fact approximations to the risk given a fixed model [17]. We review that conflating risk with uncertainty can lead to arbitrarily poor performance in a sequential decision problem [9]. We present a simple and practical solution to this problem based upon smoothed bootstrap sampling [7, 22].