<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <link href="https://bactra.org/plain-style.css" rel="stylesheet" type="text/css"> <title>Bootstrapping, and Other Resampling Methods</title> </head> <body> <P> <P></P><!-- Access to MathJax through the version I'm hosting on bactra.org !--><!-- but only IF mathmode is activated via meta_mathmode !--> <script> window.MathJax = { tex: { inlineMath: [['$','$'], ['\\(','\\)']], processEscapes: true, autoload: { color: [], colorv2: ['color'] }, packages: {'[+]': ['noerrors']} }, options: { ignoreHtmlClass: 'tex2jax_ignore', processHtmlClass: 'tex2jax_process' }, loader: { load: ['[tex]/noerrors'] }};</script> <cite><a href="http://bactra.org/notebooks">Notebooks</a></cite> <div class="text"><div class="left"><h2>Bootstrapping, and Other Resampling Methods</h2> <i><a href="faq.html#5.4">Last update</a>: 21 Apr 2025 21:17</i><br><i><a href="faq.html#5.4">First version</a>: 23 May 2010</i> <hr> <P> <P>Bootstrapping is a way of figuring out the properties of statisticalestimators (and other procedures, like hypothesis tests) by simulation. Whatwe would really like to know his how different our answers could have been, ifwe re-ran our experiment. We can't actually do this, but we can fit a model toour data and simulate from it, and see what answer we'd get from thesimulations. We can even do this from exceedingly general non-parametricestimates, like re-sampling the original data. This is a brilliant idea, andmy default way of handling the uncertainty of estimation in complex models orwith complex systems. But <a href="../weblog/652.html">having written 3500words on this for a magazine</a>, plusa <a href="http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/">textbook chapter</a>,I feel absolutely no inclination to explain myself further. <P>I most interested in resampling techniques for dependent data, and would beecstatic if I could figure out a non-parametric bootstrap for<a href="network-data-analysis.html">networks</a>. (Update: see my paper withAlden Green; the ecstasy was real but inevitably fleeting.) --- Presumably<a href="universal-prediction.html">universal prediction algorithms</a>could be used for this purpose? <P>See also: <a href="cross-validation.html">Cross-Validation</a>; <a href="confidence-sets.html">Confidence Sets, Confidence Intervals</a>; <a href="nonparametric-confidence-sets.html">Nonparametric Confidence Sets for Functions</a>; <a href="statistics.html">Statistics</a> <ul>Recommended, big picture: <li>A. C. Davison and D. V. Hinkley, <cite>Bootstrap Methods andtheir Applications</cite> <li>Bradley Efron <uL> <li>"Bootstrap Methods: Another Look at the Jackknife",<a href="http://projecteuclid.org/euclid.aos/1176344552"><cite>Annals ofStatistics</cite> <strong>7</strong> (1979): 1--26</a> [The original paper;staggeringly understandable] <li><cite>The Bootstrap, the Jackknife, and OtherResampling Plans</cite> [1982 notes volume] </ul> </ul> <ul>Recommended, close-ups: <li>Peter Bühlmann <ul> <li>"Bootstraps for Time Series",<a href="http://projecteuclid.org/euclid.ss/1023798998"><cite>Statistical Science</cite> <strong>17</strong> (2002): 52--72</a> <li>"Sieve Bootstrap with Variable Length Markov Chains for Stationary Categorical Time Series", <cite>Journal of the American Statistical Association</cite> <strong>97</strong> (2002): 443--456[<a href="http://e-collection.ethbib.ethz.ch/view/eth:24072">PDF preprint</a>] </ul> <li>Paul Doukhan, Silika Prohl, and Christian Y. Robert, "Subsamplingweakly dependent time series and application to extremes", <a href="http://arxiv.org/abs/1009.0805">arxiv:1009.0805</a> [Thanks to Dr. Prohlfor a pre-pre-print] <li>A. C. Field and A. H. Welsh, "Bootstrapping clustered data",<a href="http://dx.doi.org/10.1111/j.1467-9868.2007.00593.x"><cite>Journal ofthe Royal Statistical Society</cite> B <strong>69</strong> (2007): 369--390</a> <li>Silvia Goncalves and Halbert White, "Maximum likelihood and thebootstrap for nonlinear dynamic models",<a href="http://dx.doi.org/10.1016/S0304-4076(03)00204-5"><cite>Journal ofEconometrics</cite> <strong>119</strong> (2004): 199--219</a> <li>Peter Hall, "On Bootstrap Confidence Intervals in NonparametricRegression", <a href="http://projecteuclid.org/euclid.aos/1176348652"><cite>Annals of Statistics</cite> <strong>20</strong> (1992): 695--711</a> <li>Peter Hall and Joel Horowitz, "A simple bootstrap method for constructing nonparametric confidence bands for functions", <a href="http://dx.doi.org/10.1214/13-AOS1137"><cite>Annals of Statistics</cite> <strong>41</strong> (2013): 1892--1921</a>, <a href="http://arxiv.org/abs/1309.4864">arxiv:1309.4864</a> <li>Peter Hall and Tapabrata Maiti, "On parametric bootstrap methods for small area prediction", <a href="http://dx.doi.org/10.1111/j.1467-9868.2006.00541.x"><cite>Journal of the Royal Statistical Society</cite> B <strong>68</strong> (2006): 221--238</a> <li>Tim Hesterberg, "What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum", <a href="http://arxiv.org/abs/1411.5279">arxiv:1411.5279</a> <li>Ariel Kleiner, Ameet Talwalkar, Purnamrita Sarkar, Michael I. Jordan <ul> <li>"A Scalable Bootstrap for Massive Data", <a href="http://arxiv.org/abs/1112.5016">arxiv:1112.5016</a> <li>"The Big Data Bootstrap", <a href="http://arxiv.org/abs/1206.6415">arxiv:1206.6415</a> </ul> <li>Hans R. Künsch, "The Jackknife and the Bootstrap for GeneralStationaryObservations", <a href="http://projecteuclid.org/euclid.aos/1176347265"><cite>Annalsof Statistics</cite> <strong>17</strong> (1989): 1217--1241</a> <li>S. N. Lahiri, <cite>Resampling Methods for Dependent Data</cite>[<a href="../weblog/algae-2010-05.html#lahiri">Mini-review</a>] <li>Elizaveta Levina and Peter J. Bickel, "Texture synthesis andnonparametric resampling of randomfields", <a href="http://projecteuclid.org/euclid.aos/1162567632"><cite>Annalsof Statistics</cite><strong>34</strong> (2006): 1751--1773</a> <li>Peter Mccullagh, "Resampling and exchangeable arrays",<a href="http://projecteuclid.org/euclid.bj/1081788029"><cite>Bernoulli</cite> <strong>6</strong> (2000): 285--301</a>[Well, half-recommended. Everything he says here is right, but I think onecould construct an exactly parallel argument to show that resampling could notget correct standard errors for the mean of a stationary sequence; andof course it can't if you insist on resampling dependent data as thoughit were independent. See the papers by Owen and by Owen and Eckles.] <li>Art B. Owen, "The pigeonhole bootstrap", <a href="http://projecteuclid.org/euclid.aoas/1196438024"><cite>Annalsof Applied Statistics</cite> <strong>1</strong> (2007): 386--411</a> <li>Art B. Owen and Dean G. Eckles, "Bootstrapping data arrays of arbitrary order", <cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 895--927, <a href="http://arxiv.org/abs/1106.2125">arxiv:1106.2125</a> <li>Xianyang Zhang and Guang Cheng, "Bootstrapping High Dimensional Time Series", <a href="http://arxiv.org/abs/1406.1037">arxiv:1406.1037</a> </ul> <ul>Modesty forbids me to recommend: <li>William D. Fahy, CRS and Ryan Christopher Sullivan, "A universally applicable method of calculating confidence bands for ice nucleation spectra derived from droplet freezing experiments", <a href="https://doi.org/10.5194/amt-2022-141"><cite>Atmospheric Measurement Techniques</cite> (under review, 2022)</a> <li>Alden Green and CRS, "Bootstrapping Exchangeable Random Graphs",<a href="https://10.1214/21-EJS1896"><cite>Electronic Journal of Statistics</cite> <strong>16</strong> (2022): 1058--1095</a>, <a href="https://arxiv.org/abs/1711.00813">arxiv:1711.00813</a> <li>Robert Lunde and CRS, "Bootstrapping Generalization Error Bounds for Time Series", <a href="http://arxiv.org/abs/1711.02834">arxiv:1711.02834</a> <li>CRS <ul> <li>"TheBootstrap", <a href="http://www.americanscientist.org/issues/pub/2010/3/the-bootstrap/1"><cite>AmericanScientist</cite> <strong>98</strong> (2010): 186--190</a>[<a href="../weblog/652.html">Self-commentary</a>] <li>The chapter on the bootstrap in <cite><a href="http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/">Advanced Data Analysis from an Elementary Point of View</a></cite> </ul> </ul> <ul>To read, teaching: <li>Michael R. Chernick, <cite>Bootstrap Methods: A Practitioner'sGuide</cite> </ul> <ul>To read, learning: <li>Fumiya Akashi, Shuyang Bai, Murad S. Taqqu, "Robust Regression on Stationary Time Series: A Self-Normalized Resampling Approach",<a href="https://doi.org/10.1111/jtsa.12295"><cite>Journal of Time Series Analysis</cite> <strong>39</strong> (2018): 417--432</a> <li>Sylvain Arlot, Gilles Blanchard, and Etienne Roquain, "Some nonasymptotic results on resampling in high dimension, I: Confidence regions",<a href="http://projecteuclid.org/euclid.aos/1262271609"><cite>Annals of Statistics</cite> <strong>38</strong>(2010): 51--82</a> <li>Eytan Bakshy, Dean Eckles, "Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods", <a href="http://arxiv.org/abs/1304.7406">arxiv:1304.7406</a> <li>Raymond Chambers and Hukum Chandra, "A Random Effect Block Bootstrap for Clustered Data", <a href="http://dx.doi.org/10.1080/10618600.2012.681216"><cite>Journal of Computational and Graphical Statistics</cite> <strong>22</strong> (2013): 452--470</a> <li>Snigdhansu Chatterjee and Arup Bose, "Generalized bootstrap forestimatingequations", <a href="http://dx.doi.org/10.1214/009053604000000904"><cite>Annalsof Statistics</cite> <strong>33</strong> (2005):414--436</a>, <a href="http://arxiv.org/abs/math.ST/0504515">math.ST/0504515</a> <li>Guang Cheng and Jianhua Z. Huang, "Bootstrap consistency for general semiparametric M-estimation", <a href="http://projecteuclid.org/euclid.aos/1279638543"><cite>Annals of Statistics</cite><strong>38</strong> (2010): 2884--2915</a> <li>Andreas Christmann, Matias Salibian-Barrera, Stefan Van Aelst, "On the stability of bootstrap estimators", <a href="http://arxiv.org/abs/1111.1876">arxiv:1111.1876</a> <li>Miklos Csorgo, Masoud M Nasari, "Bootstrapped Pivots for Sample and Population Means and Distribution Functions", <a href="http://arxiv.org/abs/1307.5476">arxiv:1307.5476</a> <li>H. Dehling, O.Sh. Sharipov, M. Wendler, "Bootstrap for dependent Hilbert space-valued random variables with application to von Mises statistics", <a href="http://arxiv.org/abs/1312.3870">arxiv:1312.3870</a> <li>Herold Dehling, Martin Wendler, "Central Limit Theorem and the Bootstrap for U-Statistics of Strongly Mixing Data", <a href="http://arxiv.org/abs/0811.1888">arxiv:0811.1888</a> <li>Mathias Drton and Benjamin Williams, "Quantifying the failure of bootstrap likelihood ratio tests", <a href="http://dx.doi.org/10.1093/biomet/asr033"><cite>Biometrika</cite> <strong>98</strong> (2011): 919--934</a> <li>Bradley Efron, "Bayesian inference and the parametric bootstrap", <a href="http://dx.doi.org/10.1214/12-AOAS571"><cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 1971--1997</a> <li>Efron and Tibshirani, <cite>An Introduction to the Bootstrap</cite> <li>Mikhail Ermakov, "A moderate deviation principle for empirical bootstrap measure", <a href="http://arxiv.org/abs/1206.1459">arxiv:1206.1459</a> <li>Yanqin Fan, Qi Li and Insik Min, "A Nonparametric Bootstrap Test ofConditional Distributions", <ahref="http://dx.doi.org/10.1017/S0266466606060294"><cite>EconometricTheoy</cite> <strong>22</strong> (2006): 587--613</a> <li>Cheng-Der Fuh and Inchi Hu, "Estimation in hidden Markov models via efficient importance sampling", <cite>Bernoulli</cite><strong>13</strong> (2007): 492--513, <a href="http://arxiv.org/abs/0708.4152">arxiv:0708.4152</a> <li>Jurgen Franke, Jens-Peter Kreiss and Enno Mammen,"Bootstrap of Kernel Smoothing in Nonlinear Time Series",<a href="http://projecteuclid.org/euclid.bj/1078951087"><cite>Bernoulli</cite> <strong>8</strong> (2002): 1--37</a> <li>Axel Gandy, Patrick Rubin-Delanchy, "An algorithm to compute the power of Monte Carlo tests with guaranteed precision", <a href="http://arxiv.org/abs/1110.1248">arxiv:1110.1248</a> <li>Philip Good <ul> <li><cite>Permutation, Parametric, and Bootstrap Tests ofHypotheses</cite> <li><citE>Resampling Methods: A Practical Guide toData Analysis</cite> </ul> <li>Peter G. Hall, <cite>The Bootstrap and Edgeworth Expansion</cite>, <li>Peter Hall and Hugh Miller, "Bootstrap confidence intervals and hypothesis tests for extrema of parameters", <a href="http://dx.doi.org/10.1093/biomet/asq045"><cite>Biometrika</cite> <Strong>97</strong> (2010): 881--892</a> [E.g., looking at the largest (or smallest) of a bunch of regression coefficients] <li>Eunju Hwang, Dong Wan Shin, "Stationary bootstrapping fornon-parametric estimator of nonlinear autoregressivemodel", <a href="http://dx.doi.org/10.1111/j.1467-9892.2010.00699.x"><cite>Journalof Time Series Analysis</cite> forthcoming (2011)</a> <li>Arnold Janssen and Thorsten Pauls, "How Do Bootstrap andPermutation Tests Work?", <cite>The Annals of Statistics</cite> <strong>31</strong> (2003): 768--806 <li>Carsten Jentsch, Dimitris N. Politis and Efstathios Paparoditis, "Block Bootstrap Theory for Multivariate Integrated and Cointegrated Processes",<a href="http://dx.doi.org/10.1111/jtsa.12088"><cite>Journal of Time Series Analysis</cite> forthcoming (2014)</a> <li>Jens-Peter Kreiss, Efstathios Paparoditis, Dimitris N. Politis, "On the range of validity of the autoregressive sieve bootstrap", <cite>Annalsof Statistics</cite> <strong>39</strong> (2011): 2103--2130, <a href="http://arxiv.org/abs/12016211">arxiv:12016211</a> <li>S. N. Lahiri <ul> <li>"Asymptotic expansions for sums of block-variables under weak dependence", <a href="http://arxiv.org/abs/math/0606739">arxiv:math/0606739</a> <li>"Edgeworth expansions for studentized statistics under weak dependence", <a href="http://projecteuclid.org/euclid.aos/1262271619"><cite>Annals of Statistics</cite> <strong>38</strong> (2010): 388--434</a> </ul> <li>S. N. Lahiri, C. Spiegelman, J. Appiah, and L. Rilett, "Gap bootstrap methods for massive data sets with an application to transportation engineering", <a href="http://dx.doi.org/http://projecteuclid.org/euclid.aoas/1356629051"><cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 1552--1587</a> <li>Stephen M. S. Lee and P. Y. Lai, "Improving coverage accuracyof block bootstrap confidence intervals", <a href="http://arxiv.org/abs/0804.4361">arxiv:0804.4361</a> <li>Anne Leucht, "Degenerate U- and V-statistics under weak dependence: Asymptotic theory and bootstrap consistency", <a href="http://dx.doi.org/10.3150/11-BEJ354"><cite>Bernoulli</cite> <strong>18</strong> (2012): 552--585</a> <li>Chris J. Lloyd, "Some non-asymptotic properties of parametric bootstrap P-values in discrete models", <a href="http://dx.doi.org/10.1214/12-EJS751"><cite>Electronic Journal of Statistics</cite> <strong>6</strong> (2012): 2449--2462</a> <li>W. S. Lok and Stephen M. S. Lee, "Robustness Diagnosis for Bootstrap Inference", <a href="http://dx.doi.org/"><cite>Journal of Computational and Graphical Statistics</cite> <strong>20</strong> (2011): 448--460</a> <li>Marco Meyer and Jens-Peter Kreiss, "On the Vector Autoregressive Sieve Bootstrap", <a href="http://dx.doi.org/10.1111/jtsa.12090">Journal of Time Series Analysis</cite> forthcoming (2014)</a> <li>Michael H. Neumann, Efstathios Paparoditis, "Goodness-of-fit tests for Markovian time series models: Central limit theory and bootstrap approximations", <cite>Bernoulli</citE> <strong>14</strong> (2008): 14--46,<a href="http://arxiv.org/abs/0803.0835">arxiv:0803.0835</a> <li>Daniel J. Nordman, "A note on the stationary bootstrap'svariance", <cite>Annals of Statistics</cite> <strong>37</strong> (2009):359--370, <a href="http://arxiv.org/abs/0903.0474">arxiv:0903.0474</a> <li>Daniel J. Nordman and Soumendra N. Lahiri, "Convergence rates of empirical block length selectors for block bootstrap", <a href="http://dx.doi.org/http://projecteuclid.org/euclid.bj/1393594011"><cite>Bernoulli</cite> <strong>20</strong> (2014): 958-978</a> <li>Dimitris N. Politis, "The Impact of Bootstrap Methods onTime Series Analysis", <a href="http://projecteuclid.org/euclid.ss/1063994977"><cite>Statistical Science</cite><Strong>18</strong> (2003): 219--230</a> <li>Dimitris N. Politis, Joseph P. Romano and Michael Wolf, <cite>Subsampling</cite> <li>Zacharias Psaradakis <ul> <li>"A sieve bootstrap test for stationarity,"<a href="http://dx.doi.org/10.1016/S0167-7152(03)00012-9"><cite>Statistics andProbability Letters</cite> <strong>62</strong> (2003): 263--274</a> <li>"Blockwise bootstrap testing for stationarity",<a href="http://dx.doi.org/10.1016/j.spl.2005.09.001"><cite>Statistics andProbability Letters</cite> <strong>76</strong> (2006): 562--570</a> </ul> <li>Joseph P. Romano, Azeem M. Shaikh, "On the Uniform Asymptotic Validity of Subsampling and the Bootstrap", <a href="http://dx.doi.org/"><cite>Annals of Statistics</cite> <strong>40</strong> (2012): 2798--2822</a>, <a href="http://arxiv.org/abs/1204.2762">arxiv:1204.2762</a> <li>Matias Salibian-Barrera, Stefan van Aelst and Gert Willems,"Fast and robust bootstrap", <a href="http://dx.doi.org/10.1007/s10260-007-0048-6"><cite>Statistical Methods and Applications</cite> <strong>17</strong>(2009): 41--71</a> <li>Jun Shao and Dongsheng Tu, <citE>The Jackknife and the Bootstrap</cite> <li>Xiaofeng Shao, "The Dependent WildBootstrap", <a href="http://dx.doi.org/10.1198/jasa.2009.tm08744"><cite>Journalof the American Statistical Association</cite> <strong>105</strong> (2010):218--235</a> <li>Xiaofeng Shao and Dimitris N. Politis, "Fixed $b$ subsampling and the block bootstrap: improved confidence sets based on $p$-value calibration",<a href="http://dx.doi.org/10.1111/j.1467-9868.2012.01037.x"><cite>Journalof the Royal Statistical Society B</cite> <strong>forthcoming</strong> (2012)</a> <li>Olimjon Sh. Sharipov and Martin Wendler <uL> <li>"Bootstrap for the Sample Mean and for U-Statistics of Stationary Processes",<a href="http://arxiv.org/abs/0911.3083">arxiv:0911.3083</a> <li>"Normal Limits, Nonnormal Limits, and the Bootstrap for Quantiles of Dependent Data", <a href="http://arxiv.org/abs/1204.5633">arxiv:1204.5633</a> </ul> <li>Jan Sprenger, "Science without (parametric) models: the case of bootstrap resampling", <a href="http://dx.doi.org/10.1007/s11229-009-9567-z"><cite>Synthese</cite> <strong>180</strong> (2011): 65--76</a> <li>Johannes Tewes, "Block Bootstrap for the Empirical Process of Long‐Range Dependent Data", <a href="http://dx.doi.org/10.1111/jtsa.12256"><cite>Journal of Time Series Analysis</cite> <strong>39</strong> (2018): 28--53</a> <li>José Trashorras, Olivier Wintenberger, "Large deviations for bootstrapped empirical measures", <a href="http://arxiv.org/abs/1110.4620">arxiv:1110.4620</a> <li>Lionel Truquet, "On a nonparametric resampling scheme for Markov random fields", <a href="http://dx.doi.org/10.1214/11-EJS644"><cite>Electronic Journal of Statistics</cite> <strong>5</strong> (2011): 1503--1536</a> <li>Stefan Van Aelst and Gert Willems, "Fast and Robust Bootstrap for Multivariate Inference: The R Package FRB", <a href="http://www.jstatsoft.org/v53/i03"><cite>Journal of Statistical Software</cite> <strong>53:3</strong> (2013)</a> <li>Stanislav Volgushev, Xiaofeng Shao, "A general approach to the joint asymptotic analysis of statistics from sub-samples", <a href="http://arxiv.org/abs/1305.5618">arxiv:1305.5618</a> <li>D. Volk and M. G. Stepanov, "Resampling methods for documentclustering,"<a href="http://arxiv.org/abs/cond-mat/0109006">cond-mat/0109006</a> <li>Herwig Wendt, Patrice Abry and Stephane Jaffard, "Bootstrap forEmpirical Multifractal Analysis", <citE>IEEE Signal Processing Magazine</cite>July 2007, pp. 38--48 [+ technical papers by these authors] <li>Michael Wolf and Dan Wunderli, "Bootstrap Joint Prediction Regions", <a href="http://dx.doi.org/10.1111/jtsa.12099"><cite>Journal of Time Series Analysis</cite> <strong>forthcoming</strong> (2014)</a> <li>Guosheng Yin and Yanyuan Ma, "Pearson-type goodness-of-fit test with bootstrap maximum likelihood estimation", <a href="http://dx.doi.org/0.1214/13-EJS773"><cite>Electronic Journal of Statistics</cite> <strong>7</strong> (2013): 412--427</a> <li>Ting Zhang, Hwai-Chung Ho, Martin Wendler, Wei Biao Wu, "Block Sampling under Strong Dependence", <a href="http://arxiv.org/abs/1312.5807">arxiv:1312.5807</a> <li>Abdelhak M. Zoubir and D. Robert Iskander, <citE>BootstrapTechniques for Signal Processing</cite> </ul> </p> </div></div> <hr> <div class="footer"><P><a href="http://bactra.org/notebooks/bootstrap.html">permanent link for thisnote</a> <a href="http://bactra.org/notebooks/bootstrap.rss">RSS feed for thisnote</a></P></div><div class="foot"> <center> <cite><a href="http://bactra.org/notebooks">Notebooks</a></cite>: <a href="http://www.blosxom.com/">Powered by Blosxom</a> </center> </div> </body></html>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<link href="https://bactra.org/plain-style.css" rel="stylesheet" type="text/css">
<title>Bootstrapping, and Other Resampling Methods</title>
</head>
<body>
<P>
<P></P><!-- Access to MathJax through the version I'm hosting on bactra.org !-->
<!-- but only IF mathmode is activated via meta_mathmode !-->
<script>
window.MathJax = {
tex: {
inlineMath: [['$','$'], ['\\(','\\)']],
processEscapes: true,
autoload: {
color: [],
colorv2: ['color']
},
packages: {'[+]': ['noerrors']}
},
options: {
ignoreHtmlClass: 'tex2jax_ignore',
processHtmlClass: 'tex2jax_process'
},
loader: {
load: ['[tex]/noerrors']
}
};
</script>
<cite><a href="http://bactra.org/notebooks">Notebooks</a></cite>
<div class="text">
<div class="left">
<h2>Bootstrapping, and Other Resampling Methods</h2>
<i><a href="faq.html#5.4">Last update</a>: 21 Apr 2025 21:17</i>
<br><i><a href="faq.html#5.4">First version</a>: 23 May 2010</i>
<hr>
<P>
<P>Bootstrapping is a way of figuring out the properties of statistical
estimators (and other procedures, like hypothesis tests) by simulation. What
we would really like to know his how different our answers could have been, if
we re-ran our experiment. We can't actually do this, but we can fit a model to
our data and simulate from it, and see what answer we'd get from the
simulations. We can even do this from exceedingly general non-parametric
estimates, like re-sampling the original data. This is a brilliant idea, and
my default way of handling the uncertainty of estimation in complex models or
with complex systems. But <a href="../weblog/652.html">having written 3500
words on this for a magazine</a>, plus
a <a href="http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/">textbook chapter</a>,
I feel absolutely no inclination to explain myself further.
<P>I most interested in resampling techniques for dependent data, and would be
ecstatic if I could figure out a non-parametric bootstrap for
<a href="network-data-analysis.html">networks</a>. (Update: see my paper with
Alden Green; the ecstasy was real but inevitably fleeting.) --- Presumably
<a href="universal-prediction.html">universal prediction algorithms</a>
could be used for this purpose?
<P>See also:
<a href="cross-validation.html">Cross-Validation</a>;
<a href="confidence-sets.html">Confidence Sets, Confidence Intervals</a>;
<a href="nonparametric-confidence-sets.html">Nonparametric Confidence Sets for Functions</a>;
<a href="statistics.html">Statistics</a>
<ul>Recommended, big picture:
<li>A. C. Davison and D. V. Hinkley, <cite>Bootstrap Methods and
their Applications</cite>
<li>Bradley Efron
<uL>
<li>"Bootstrap Methods: Another Look at the Jackknife",
<a href="http://projecteuclid.org/euclid.aos/1176344552"><cite>Annals of
Statistics</cite> <strong>7</strong> (1979): 1--26</a> [The original paper;
staggeringly understandable]
<li><cite>The Bootstrap, the Jackknife, and Other
Resampling Plans</cite> [1982 notes volume]
</ul>
</ul>
<ul>Recommended, close-ups:
<li>Peter Bühlmann
<ul>
<li>"Bootstraps for Time Series",
<a href="http://projecteuclid.org/euclid.ss/1023798998"><cite>Statistical Science</cite> <strong>17</strong> (2002): 52--72</a>
<li>"Sieve Bootstrap with Variable Length Markov Chains for Stationary Categorical Time Series", <cite>Journal of the American Statistical Association</cite> <strong>97</strong> (2002): 443--456
[<a href="http://e-collection.ethbib.ethz.ch/view/eth:24072">PDF preprint</a>]
</ul>
<li>Paul Doukhan, Silika Prohl, and Christian Y. Robert, "Subsampling
weakly dependent time series and application to extremes", <a href="http://arxiv.org/abs/1009.0805">arxiv:1009.0805</a> [Thanks to Dr. Prohl
for a pre-pre-print]
<li>A. C. Field and A. H. Welsh, "Bootstrapping clustered data",
<a href="http://dx.doi.org/10.1111/j.1467-9868.2007.00593.x"><cite>Journal of
the Royal Statistical Society</cite> B <strong>69</strong> (2007): 369--390</a>
<li>Silvia Goncalves and Halbert White, "Maximum likelihood and the
bootstrap for nonlinear dynamic models",
<a href="http://dx.doi.org/10.1016/S0304-4076(03)00204-5"><cite>Journal of
Econometrics</cite> <strong>119</strong> (2004): 199--219</a>
<li>Peter Hall, "On Bootstrap Confidence Intervals in Nonparametric
Regression", <a href="http://projecteuclid.org/euclid.aos/1176348652"><cite>Annals of Statistics</cite> <strong>20</strong> (1992): 695--711</a>
<li>Peter Hall and Joel Horowitz, "A simple bootstrap method for constructing nonparametric confidence bands for functions", <a href="http://dx.doi.org/10.1214/13-AOS1137"><cite>Annals of Statistics</cite> <strong>41</strong> (2013): 1892--1921</a>, <a href="http://arxiv.org/abs/1309.4864">arxiv:1309.4864</a>
<li>Peter Hall and Tapabrata Maiti, "On parametric bootstrap methods for small area prediction", <a href="http://dx.doi.org/10.1111/j.1467-9868.2006.00541.x"><cite>Journal of the Royal Statistical Society</cite> B <strong>68</strong> (2006): 221--238</a>
<li>Tim Hesterberg, "What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum", <a href="http://arxiv.org/abs/1411.5279">arxiv:1411.5279</a>
<li>Ariel Kleiner, Ameet Talwalkar, Purnamrita Sarkar, Michael I. Jordan
<ul>
<li>"A Scalable Bootstrap for Massive Data", <a href="http://arxiv.org/abs/1112.5016">arxiv:1112.5016</a>
<li>"The Big Data Bootstrap", <a href="http://arxiv.org/abs/1206.6415">arxiv:1206.6415</a>
</ul>
<li>Hans R. Künsch, "The Jackknife and the Bootstrap for General
Stationary
Observations", <a href="http://projecteuclid.org/euclid.aos/1176347265"><cite>Annals
of Statistics</cite> <strong>17</strong> (1989): 1217--1241</a>
<li>S. N. Lahiri, <cite>Resampling Methods for Dependent Data</cite>
[<a href="../weblog/algae-2010-05.html#lahiri">Mini-review</a>]
<li>Elizaveta Levina and Peter J. Bickel, "Texture synthesis and
nonparametric resampling of random
fields", <a href="http://projecteuclid.org/euclid.aos/1162567632"><cite>Annals
of Statistics</cite>
<strong>34</strong> (2006): 1751--1773</a>
<li>Peter Mccullagh, "Resampling and exchangeable arrays",
<a href="http://projecteuclid.org/euclid.bj/1081788029"><cite>Bernoulli</cite> <strong>6</strong> (2000): 285--301</a>
[Well, half-recommended. Everything he says here is right, but I think one
could construct an exactly parallel argument to show that resampling could not
get correct standard errors for the mean of a stationary sequence; and
of course it can't if you insist on resampling dependent data as though
it were independent. See the papers by Owen and by Owen and Eckles.]
<li>Art B. Owen, "The pigeonhole bootstrap", <a href="http://projecteuclid.org/euclid.aoas/1196438024"><cite>Annals
of Applied Statistics</cite> <strong>1</strong> (2007): 386--411</a>
<li>Art B. Owen and Dean G. Eckles, "Bootstrapping data arrays of arbitrary order", <cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 895--927, <a href="http://arxiv.org/abs/1106.2125">arxiv:1106.2125</a>
<li>Xianyang Zhang and Guang Cheng, "Bootstrapping High Dimensional Time Series", <a href="http://arxiv.org/abs/1406.1037">arxiv:1406.1037</a>
</ul>
<ul>Modesty forbids me to recommend:
<li>William D. Fahy, CRS and Ryan Christopher Sullivan, "A universally applicable method of calculating confidence bands for ice nucleation spectra derived from droplet freezing experiments", <a href="https://doi.org/10.5194/amt-2022-141"><cite>Atmospheric Measurement Techniques</cite> (under review, 2022)</a>
<li>Alden Green and CRS, "Bootstrapping Exchangeable Random Graphs",
<a href="https://10.1214/21-EJS1896"><cite>Electronic Journal of Statistics</cite> <strong>16</strong> (2022): 1058--1095</a>, <a href="https://arxiv.org/abs/1711.00813">arxiv:1711.00813</a>
<li>Robert Lunde and CRS, "Bootstrapping Generalization Error Bounds for Time Series", <a href="http://arxiv.org/abs/1711.02834">arxiv:1711.02834</a>
<li>CRS
<ul>
<li>"The
Bootstrap", <a href="http://www.americanscientist.org/issues/pub/2010/3/the-bootstrap/1"><cite>American
Scientist</cite> <strong>98</strong> (2010): 186--190</a>
[<a href="../weblog/652.html">Self-commentary</a>]
<li>The chapter on the bootstrap in <cite><a href="http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/">Advanced Data Analysis from an Elementary Point of View</a></cite>
</ul>
</ul>
<ul>To read, teaching:
<li>Michael R. Chernick, <cite>Bootstrap Methods: A Practitioner's
Guide</cite>
</ul>
<ul>To read, learning:
<li>Fumiya Akashi, Shuyang Bai, Murad S. Taqqu, "Robust Regression on Stationary Time Series: A Self-Normalized Resampling Approach",
<a href="https://doi.org/10.1111/jtsa.12295"><cite>Journal of Time Series Analysis</cite> <strong>39</strong> (2018): 417--432</a>
<li>Sylvain Arlot, Gilles Blanchard, and Etienne Roquain, "Some nonasymptotic results on resampling in high dimension, I: Confidence regions",
<a href="http://projecteuclid.org/euclid.aos/1262271609"><cite>Annals of Statistics</cite> <strong>38</strong>
(2010): 51--82</a>
<li>Eytan Bakshy, Dean Eckles, "Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods", <a href="http://arxiv.org/abs/1304.7406">arxiv:1304.7406</a>
<li>Raymond Chambers and Hukum Chandra, "A Random Effect Block Bootstrap for Clustered Data", <a href="http://dx.doi.org/10.1080/10618600.2012.681216"><cite>Journal of Computational and Graphical Statistics</cite> <strong>22</strong> (2013): 452--470</a>
<li>Snigdhansu Chatterjee and Arup Bose, "Generalized bootstrap for
estimating
equations", <a href="http://dx.doi.org/10.1214/009053604000000904"><cite>Annals
of Statistics</cite> <strong>33</strong> (2005):
414--436</a>, <a href="http://arxiv.org/abs/math.ST/0504515">math.ST/0504515</a>
<li>Guang Cheng and Jianhua Z. Huang, "Bootstrap consistency for general semiparametric M-estimation", <a href="http://projecteuclid.org/euclid.aos/1279638543"><cite>Annals of Statistics</cite>
<strong>38</strong> (2010): 2884--2915</a>
<li>Andreas Christmann, Matias Salibian-Barrera, Stefan Van Aelst, "On the stability of bootstrap estimators", <a href="http://arxiv.org/abs/1111.1876">arxiv:1111.1876</a>
<li>Miklos Csorgo, Masoud M Nasari, "Bootstrapped Pivots for Sample and Population Means and Distribution Functions", <a href="http://arxiv.org/abs/1307.5476">arxiv:1307.5476</a>
<li>H. Dehling, O.Sh. Sharipov, M. Wendler, "Bootstrap for dependent Hilbert space-valued random variables with application to von Mises statistics", <a href="http://arxiv.org/abs/1312.3870">arxiv:1312.3870</a>
<li>Herold Dehling, Martin Wendler, "Central Limit Theorem and the Bootstrap for U-Statistics of Strongly Mixing Data", <a href="http://arxiv.org/abs/0811.1888">arxiv:0811.1888</a>
<li>Mathias Drton and Benjamin Williams, "Quantifying the failure of bootstrap likelihood ratio tests", <a href="http://dx.doi.org/10.1093/biomet/asr033
"><cite>Biometrika</cite> <strong>98</strong> (2011): 919--934</a>
<li>Bradley Efron, "Bayesian inference and the parametric bootstrap", <a href="http://dx.doi.org/10.1214/12-AOAS571"><cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 1971--1997</a>
<li>Efron and Tibshirani, <cite>An Introduction to the Bootstrap</cite>
<li>Mikhail Ermakov, "A moderate deviation principle for empirical bootstrap measure", <a href="http://arxiv.org/abs/1206.1459">arxiv:1206.1459</a>
<li>Yanqin Fan, Qi Li and Insik Min, "A Nonparametric Bootstrap Test of
Conditional Distributions", <a
href="http://dx.doi.org/10.1017/S0266466606060294"><cite>Econometric
Theoy</cite> <strong>22</strong> (2006): 587--613</a>
<li>Cheng-Der Fuh and Inchi Hu, "Estimation in hidden Markov models via efficient importance sampling", <cite>Bernoulli</cite>
<strong>13</strong> (2007): 492--513, <a href="http://arxiv.org/abs/0708.4152">arxiv:0708.4152</a>
<li>Jurgen Franke, Jens-Peter Kreiss and Enno Mammen,
"Bootstrap of Kernel Smoothing in Nonlinear Time Series",
<a href="http://projecteuclid.org/euclid.bj/1078951087"><cite>Bernoulli</cite> <strong>8</strong> (2002): 1--37</a>
<li>Axel Gandy, Patrick Rubin-Delanchy, "An algorithm to compute the power of Monte Carlo tests with guaranteed precision", <a href="http://arxiv.org/abs/1110.1248">arxiv:1110.1248</a>
<li>Philip Good
<ul>
<li><cite>Permutation, Parametric, and Bootstrap Tests of
Hypotheses</cite>
<li><citE>Resampling Methods: A Practical Guide to
Data Analysis</cite>
</ul>
<li>Peter G. Hall, <cite>The Bootstrap and Edgeworth Expansion</cite>,
<li>Peter Hall and Hugh Miller, "Bootstrap confidence intervals and hypothesis tests for extrema of parameters", <a href="http://dx.doi.org/10.1093/biomet/asq045"><cite>Biometrika</cite> <Strong>97</strong> (2010): 881--892</a> [E.g., looking at the largest (or smallest) of a bunch of regression coefficients]
<li>Eunju Hwang, Dong Wan Shin, "Stationary bootstrapping for
non-parametric estimator of nonlinear autoregressive
model", <a href="http://dx.doi.org/10.1111/j.1467-9892.2010.00699.x"><cite>Journal
of Time Series Analysis</cite> forthcoming (2011)</a>
<li>Arnold Janssen and Thorsten Pauls, "How Do Bootstrap and
Permutation Tests Work?", <cite>The Annals of Statistics</cite> <strong>31</strong> (2003): 768--806
<li>Carsten Jentsch, Dimitris N. Politis and Efstathios Paparoditis, "Block Bootstrap Theory for Multivariate Integrated and Cointegrated Processes",
<a href="http://dx.doi.org/10.1111/jtsa.12088"><cite>Journal of Time Series Analysis</cite> forthcoming (2014)</a>
<li>Jens-Peter Kreiss, Efstathios Paparoditis, Dimitris N. Politis, "On the range of validity of the autoregressive sieve bootstrap", <cite>Annals
of Statistics</cite> <strong>39</strong> (2011): 2103--2130, <a href="http://arxiv.org/abs/12016211">arxiv:12016211</a>
<li>S. N. Lahiri
<ul>
<li>"Asymptotic expansions for sums of block-variables under weak dependence", <a href="http://arxiv.org/abs/math/0606739">arxiv:math/0606739</a>
<li>"Edgeworth expansions for studentized statistics under weak dependence", <a href="http://projecteuclid.org/euclid.aos/1262271619"><cite>Annals of Statistics</cite> <strong>38</strong> (2010): 388--434</a>
</ul>
<li>S. N. Lahiri, C. Spiegelman, J. Appiah, and L. Rilett, "Gap bootstrap methods for massive data sets with an application to transportation engineering", <a href="http://dx.doi.org/http://projecteuclid.org/euclid.aoas/1356629051"><cite>Annals of Applied Statistics</cite> <strong>6</strong> (2012): 1552--1587</a>
<li>Stephen M. S. Lee and P. Y. Lai, "Improving coverage accuracy
of block bootstrap confidence intervals", <a href="http://arxiv.org/abs/0804.4361">arxiv:0804.4361</a>
<li>Anne Leucht, "Degenerate U- and V-statistics under weak dependence: Asymptotic theory and bootstrap consistency", <a href="http://dx.doi.org/10.3150/11-BEJ354"><cite>Bernoulli</cite> <strong>18</strong> (2012): 552--585</a>
<li>Chris J. Lloyd, "Some non-asymptotic properties of parametric bootstrap P-values in discrete models", <a href="http://dx.doi.org/10.1214/12-EJS751"><cite>Electronic Journal of Statistics</cite> <strong>6</strong> (2012): 2449--2462</a>
<li>W. S. Lok and Stephen M. S. Lee, "Robustness Diagnosis for Bootstrap Inference", <a href="http://dx.doi.org/"><cite>Journal of Computational and Graphical Statistics</cite> <strong>20</strong> (2011): 448--460</a>
<li>Marco Meyer and Jens-Peter Kreiss, "On the Vector Autoregressive Sieve Bootstrap", <a href="http://dx.doi.org/10.1111/jtsa.12090">Journal of Time Series Analysis</cite> forthcoming (2014)</a>
<li>Michael H. Neumann, Efstathios Paparoditis, "Goodness-of-fit tests for Markovian time series models: Central limit theory and bootstrap approximations", <cite>Bernoulli</citE> <strong>14</strong> (2008): 14--46,
<a href="http://arxiv.org/abs/0803.0835">arxiv:0803.0835</a>
<li>Daniel J. Nordman, "A note on the stationary bootstrap's
variance", <cite>Annals of Statistics</cite> <strong>37</strong> (2009):
359--370, <a href="http://arxiv.org/abs/0903.0474">arxiv:0903.0474</a>
<li>Daniel J. Nordman and Soumendra N. Lahiri, "Convergence rates of empirical block length selectors for block bootstrap", <a href="http://dx.doi.org/http://projecteuclid.org/euclid.bj/1393594011"><cite>Bernoulli</cite> <strong>20</strong> (2014): 958-978</a>
<li>Dimitris N. Politis, "The Impact of Bootstrap Methods on
Time Series Analysis", <a href="http://projecteuclid.org/euclid.ss/1063994977"><cite>Statistical Science</cite>
<Strong>18</strong> (2003): 219--230</a>
<li>Dimitris N. Politis, Joseph P. Romano and Michael Wolf, <cite>Subsampling</cite>
<li>Zacharias Psaradakis
<ul>
<li>"A sieve bootstrap test for stationarity,"
<a href="http://dx.doi.org/10.1016/S0167-7152(03)00012-9"><cite>Statistics and
Probability Letters</cite> <strong>62</strong> (2003): 263--274</a>
<li>"Blockwise bootstrap testing for stationarity",
<a href="http://dx.doi.org/10.1016/j.spl.2005.09.001"><cite>Statistics and
Probability Letters</cite> <strong>76</strong> (2006): 562--570</a>
</ul>
<li>Joseph P. Romano, Azeem M. Shaikh, "On the Uniform Asymptotic Validity of Subsampling and the Bootstrap", <a href="http://dx.doi.org/"><cite>Annals of Statistics</cite> <strong>40</strong> (2012): 2798--2822</a>, <a href="http://arxiv.org/abs/1204.2762">arxiv:1204.2762</a>
<li>Matias Salibian-Barrera, Stefan van Aelst and Gert Willems,
"Fast and robust bootstrap", <a href="http://dx.doi.org/10.1007/s10260-007-0048-6"><cite>Statistical Methods and Applications</cite> <strong>17</strong>
(2009): 41--71</a>
<li>Jun Shao and Dongsheng Tu, <citE>The Jackknife and the Bootstrap</cite>
<li>Xiaofeng Shao, "The Dependent Wild
Bootstrap", <a href="http://dx.doi.org/10.1198/jasa.2009.tm08744"><cite>Journal
of the American Statistical Association</cite> <strong>105</strong> (2010):
218--235</a>
<li>Xiaofeng Shao and Dimitris N. Politis, "Fixed $b$ subsampling and the block bootstrap: improved confidence sets based on $p$-value calibration",
<a href="http://dx.doi.org/10.1111/j.1467-9868.2012.01037.x"><cite>Journal
of the Royal Statistical Society B</cite> <strong>forthcoming</strong> (2012)</a>
<li>Olimjon Sh. Sharipov and Martin Wendler
<uL>
<li>"Bootstrap for the Sample Mean and for U-Statistics of Stationary Processes",
<a href="http://arxiv.org/abs/0911.3083">arxiv:0911.3083</a>
<li>"Normal Limits, Nonnormal Limits, and the Bootstrap for Quantiles of Dependent Data", <a href="http://arxiv.org/abs/1204.5633">arxiv:1204.5633</a>
</ul>
<li>Jan Sprenger, "Science without (parametric) models: the case of bootstrap resampling", <a href="http://dx.doi.org/10.1007/s11229-009-9567-z"><cite>Synthese</cite> <strong>180</strong> (2011): 65--76</a>
<li>Johannes Tewes, "Block Bootstrap for the Empirical Process of Long‐Range Dependent Data", <a href="http://dx.doi.org/10.1111/jtsa.12256"><cite>Journal of Time Series Analysis</cite> <strong>39</strong> (2018): 28--53</a>
<li>José Trashorras, Olivier Wintenberger, "Large deviations for bootstrapped empirical measures", <a href="http://arxiv.org/abs/1110.4620">arxiv:1110.4620</a>
<li>Lionel Truquet, "On a nonparametric resampling scheme for Markov random fields", <a href="http://dx.doi.org/10.1214/11-EJS644"><cite>Electronic Journal of Statistics</cite> <strong>5</strong> (2011): 1503--1536</a>
<li>Stefan Van Aelst and Gert Willems, "Fast and Robust Bootstrap for Multivariate Inference: The R Package FRB", <a href="http://www.jstatsoft.org/v53/i03"><cite>Journal of Statistical Software</cite> <strong>53:3</strong> (2013)</a>
<li>Stanislav Volgushev, Xiaofeng Shao, "A general approach to the joint asymptotic analysis of statistics from sub-samples", <a href="http://arxiv.org/abs/1305.5618">arxiv:1305.5618</a>
<li>D. Volk and M. G. Stepanov, "Resampling methods for document
clustering,"
<a href="http://arxiv.org/abs/cond-mat/0109006">cond-mat/0109006</a>
<li>Herwig Wendt, Patrice Abry and Stephane Jaffard, "Bootstrap for
Empirical Multifractal Analysis", <citE>IEEE Signal Processing Magazine</cite>
July 2007, pp. 38--48 [+ technical papers by these authors]
<li>Michael Wolf and Dan Wunderli, "Bootstrap Joint Prediction Regions", <a href="http://dx.doi.org/10.1111/jtsa.12099"><cite>Journal of Time Series Analysis</cite> <strong>forthcoming</strong> (2014)</a>
<li>Guosheng Yin and Yanyuan Ma, "Pearson-type goodness-of-fit test with bootstrap maximum likelihood estimation", <a href="http://dx.doi.org/0.1214/13-EJS773"><cite>Electronic Journal of Statistics</cite> <strong>7</strong> (2013): 412--427</a>
<li>Ting Zhang, Hwai-Chung Ho, Martin Wendler, Wei Biao Wu, "Block Sampling under Strong Dependence", <a href="http://arxiv.org/abs/1312.5807">arxiv:1312.5807</a>
<li>Abdelhak M. Zoubir and D. Robert Iskander, <citE>Bootstrap
Techniques for Signal Processing</cite>
</ul>
</p>
</div>
</div>
<hr>
<div class="footer"><P><a href="http://bactra.org/notebooks/bootstrap.html">permanent link for this
note</a> <a href="http://bactra.org/notebooks/bootstrap.rss">RSS feed for this
note</a></P></div>
<div class="foot">
<center>
<cite><a href="http://bactra.org/notebooks">Notebooks</a></cite>:
<a href="http://www.blosxom.com/">Powered by Blosxom</a>
</center>
</div>
</body></html>