Bayesian analysis; Posterior distribution; Prior distribution; Stein's method; Statistics and Probability; Statistics, Probability and Uncertainty
Abstract :
[en] In this paper, we propose tight upper and lower bounds for the Wasserstein distance between any two univariate continuous distributions with probability densities p1 and p2 having nested supports. These explicit bounds are expressed in terms of the derivative of the likelihood ratio p1/p2 as well as the Stein kernel τ1 of p1. The method of proof relies on a new variant of Stein's method which manipulates Stein operators. We give several applications of these bounds. Our main application is in Bayesian statistics: we derive explicit data-driven bounds on the Wasserstein distance between the posterior distribution based on a given prior and the no-prior posterior based uniquely on the sampling distribution. This is the first finite sample result confirming the well-known fact that with well-identified parameters and large sample sizes, reasonable choices of prior distributions will have only minor effects on posterior inferences if the data are benign.
Disciplines :
Mathematics
Author, co-author :
LEY, Christophe ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Mathematics (DMATH) ; Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, Belgium
Reinert, Gesine; Department of Statistics, University of Oxford, Oxford, United Kingdom
Swan, Yvik; Département de Mathématique, Université de Liège, Liège, Belgium
External co-authors :
yes
Language :
English
Title :
Distances between nested densities and a measure of the impact of the prior in Bayesian statistics
Supported by EPSRC Grant EP/K032402/I and by the Keble College Advanced Studies Centre. Supported by the IAP Research Network P7/06 of the Belgian State (Belgian Science Policy). The authors thank the three referees for their very careful reading of the paper and the many pertinent remarks and corrections they suggested. Christophe Ley and Yvik Swan also wish to thank Keble College for logistic support during an important writing stage of this paper.
AZZALINI, A. (1985). A class of distributions which includes the normal ones. Scand. J. Statist. 12 171-178. MR0808153
CHEN, L. H. Y., GOLDSTEIN, L. and SHAO, Q.-M. (2011). Normal Approximation by Stein's Method. Probability and Its Applications (New York). Springer, Heidelberg. MR2732624
CHWIALKOWSKI, K., STRATHMANN, H. and GRETTON, A. (2016). A kernel test of goodness of fit. Preprint. Available at arXiv:1602.02964v3.
DIACONIS, P. and FREEDMAN, D. (1986). On inconsistent Bayes estimates of location. Ann. Statist. 14 68-87. MR0829556
DIACONIS, P. and FREEDMAN, D. (1986). On the consistency of Bayes estimates. Ann. Statist. 14 1-67. MR0829555
DÖBLER, C. (2015). Stein's method of exchangeable pairs for the beta distribution and generalizations. Electron. J. Probab. 20 109. MR3418541
DÖBLER, C. (2015). Stein's method for the half-normal distribution with applications to limit theorems related to the simple symmetric random walk. ALEA Lat. Am. J. Probab. Math. Stat. 12 171-191. MR3343481
EDEN, R. and VÍQUEZ, J. (2015). Nourdin-Peccati analysis on Wiener and Wiener-Poisson space for general distributions. Stochastic Process. Appl. 125 182-216. MR3274696
EFRON, B. (1981). Nonparametric standard errors and confidence intervals. Canad. J. Statist. 9 139-172. MR0640014
EICHELSBACHER, P. and THÄLE, C. (2015). Malliavin-Stein method for variance-gamma approximation on Wiener space. Electron. J. Probab. 20 123. MR3425543
GAUNT, R. E. (2014). Variance-gamma approximation via Stein's method. Electron. J. Probab. 19 38. MR3194737
GOLDSTEIN, L. and REINERT, G. (2005). Distributional transformations, orthogonal polynomials, and Stein characterizations. J. Theoret. Probab. 18 237-260. MR2132278
GORHAM, J. and MACKEY, L. (2015). Measuring sample quality with Stein's method. Adv. Neural Inf. Process. Syst. 226-234.
GORHAM, J. and MACKEY, L. (2016). Multivariate Stein factors for strongly log-concave distributions. Electron. Commun. Probab. 21.
HALLIN, M. and LEY, C. (2014). Skew-symmetric distributions and Fisher information: The double sin of the skew-normal. Bernoulli 20 1432-1453. MR3217449
KARLIN, S. and RUBIN, H. (1956). Distributions possessing a monotone likelihood ratio. J. Amer. Statist. Assoc. 51 637-643. MR0104303
LEY, C., REINERT, G. and SWAN, Y. (2016). Stein's method for comparison of univariate distributions. Preprint. Available at arXiv:1408.2998.
LEY, C. and SWAN, Y. (2013). Local Pinsker inequalities via Stein's discrete density approach. IEEE Trans. Inform. Theory 59 5584-5591. MR3096942
LEY, C. and SWAN, Y. (2013). Stein's density approach and information inequalities. Electron. Commun. Probab. 18 7. MR3019670
LEY, C. and SWAN, Y. (2016). Parametric Stein operators and variance bounds. Braz. J. Probab. Stat. 30 171-195. MR3481100
NOURDIN, I. and PECCATI, G. (2012). Normal Approximations with Malliavin Calculus: From Stein's Method to Universality. Cambridge Tracts in Mathematics 192. Cambridge Univ. Press, Cambridge. MR2962301
NOURDIN, I., PECCATI, G. and SWAN, Y. (2014). Entropy and the fourth moment phenomenon. J. Funct. Anal. 266 3170-3207. MR3158721
NOURDIN, I., PECCATI, G. and SWAN, Y. (2014). Integration by parts and representation of information functionals. 2014 IEEE International Symposium on Information Theory (ISIT) 2217-2221.
OATES, C. J., GIROLAMI, M. and CHOPIN, N. (2016). Control funtionals for Monte Carlo integration. J. R. Stat. Soc. Ser. B. Stat. Methodol. To appear. DOI:10.1111/rssb.12185.
PIKE, J. and REN, H. (2014). Stein's method and the Laplace distribution. ALEA Lat. Am. J. Probab. Math. Stat. 11 571-587. MR3283586
ROSS, N. (2011). Fundamentals of Stein's method. Probab. Surv. 8 210-293. MR2861132
ROSS, S. M. (1996). Stochastic Processes, 2nd ed. Wiley Series in Probability and Statistics: Probability and Statistics. Wiley, New York. MR1373653
SHAKED, M. and SHANTHIKUMAR, J. G. (2007). Stochastic Orders. Springer Series in Statistics. Springer, New York. MR2265633
STEIN, C. (1965). Approximation of improper prior measures by prior probability measures. In Proc. Internat. Res. Sem., Statist. Lab., Univ. California, Berkeley, Calif., 1963 217-240. Springer, New York. MR0199937
STEIN, C. (1986). Approximate Computation of Expectations. Institute of Mathematical Statistics Lecture Notes - Monograph Series 7. IMS, Hayward, CA. MR0882007
STEIN, C., DIACONIS, P., HOLMES, S. and REINERT, G. (2004). Use of exchangeable pairs in the analysis of simulations. In Stein's Method: Expository Lectures and Applications. Institute of Mathematical Statistics Lecture Notes - Monograph Series 46 1-26. IMS, Beachwood, OH. MR2118600
VALLENDER, S. (1974). Calculation of the Wasserstein distance between probability distributions on the line. Theory Probab. Appl. 18 784-786.
VILLANI, C. (2009). Optimal Transport: Old and New. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 338. Springer, Berlin. MR2459454
