06重抽样算法

Lingfeng2025-05-21

06重抽样算法

1. Bootstrap 估计

1.1 估计量偏差的 Bootstrap 估计

Definition (估计量偏差的 Bootstrap 估计)

从样本 $X_{1},X_{2},\dots,X_{n}$ 中有放回地产生数据 $X^{*}_{11},X^{*}_{12},\dots,X^{*}_{1n}$ ，得到 $\theta$ 的估计并记为 $\hat{\theta}_{1}^{*}= \hat{\theta}_{1}^{*}(X^{*}_{11},X^{*}_{12},\dots,X^{*}_{1n})$
重复上述步骤 $m$ 次，相应估计分别记为 $\hat{\theta}^{*}_{1},\hat{\theta}^{*}_{2},\dots,\hat{\theta}^{*}_{m}$ ，以及基于样本 $X_{1},X_{2},\dots,X_{n}$ 得到估计 $\hat{\theta}$
计算偏差的 Bootstrap 估计 $\widehat{\text{Bias}}(\hat{\theta})=\frac{1}{m}\sum_{i = 1}^{m} \hat{\theta}_{i}^{*}-\hat{\theta}$ 此时经偏差修正的参数 $\theta$ 的估计量为 $\tilde{\theta}=\hat{\theta}-\widehat{\text{Bias}}(\hat{\theta})$

Theorem (样本方差偏差的 Bootstrap 估计)

假设 $X_{1},X_{2},\dots,X_{n}$ 来自方差为 $\sigma^{2}$ 的分布，则 $\sigma^{2}$ 估计为

\hat{\sigma}^{2}=\frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}

请计算

\text{Bias}(\hat{\sigma}^{2})

，并判断

\widehat{\text{Bias}}(\hat{\sigma}^{2})

是否为无偏估计。

由定义

\begin{align*} \text{Bias}(\hat{\sigma}^{2}) & =E(\hat{\sigma}^{2})-\sigma^{2} \\ & = \frac{n-1}{n}\sigma^{2}-\sigma^{2} \\ & = -\frac{\sigma^{2}}{n} \end{align*}

注意到对于

X_{1i}^{*}

，满足

\begin{align*} E(X_{1i}^{*}\mid X_{1},X_{2},\dots,X_{n}) & =\bar{X} \\ Var(X_{1i}^{*}\mid X_{1},X_{2},\dots,X_{n}) & =\frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

从而

\begin{align*} E(\hat{\sigma}_{1}^{2*}\mid X_{1},X_{2},\dots,X_{n}) & =E\left( \frac{1}{n}\sum_{i = 1}^{n} (X_{1i}^{*}-\bar{X}^{*}_{1})^{2} \mid X_{1},X_{2},\dots,X_{n}\right) \\ & = \frac{n-1}{n} Var(X_{11}^{*}\mid X_{1},X_{2},\dots,X_{n}) \\ & = \frac{n-1}{n^{2}}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

因此

\begin{align*} E(\widehat{\text{Bias}}(\hat{\sigma}^{2})\mid X_{1},X_{2},\dots,X_{n}) & =E\left( \frac{1}{m}\sum_{i = 1}^{m} \hat{\sigma}^{2*}_{i}-\hat{\sigma}^{2} \mid X_{1},X_{2},\dots,X_{n}\right) \\ & = E(\hat{\sigma}_{1}^{2*}\mid X_{1},X_{2},\dots,X_{n})-\hat{\sigma}^{2} \\ & = \frac{n-1}{n^{2}}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}- \frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \\ & = -\frac{1}{n^{2}}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

最后得到

E(\widehat{\text{Bias}}(\hat{\sigma}^{2}))=-\frac{n-1}{n^{2}}\sigma^{2}\neq-\frac{1}{n}\sigma^{2}=\text{Bias}(\hat{\sigma}^{2})

Example (样本无偏方差的 Bootstrap 估计)

考虑 $\hat{\sigma}^{2}$ 估计为

\hat{\sigma}^{2}=\frac{1}{n-1}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}

计算

E(\widehat{\text{Bias}}(\hat{\sigma}^{2}))

考虑

\begin{align*} E(\hat{\sigma}^{2*}_{1}\mid X_{1},X_{2},\dots,X_{n}) & = E\left( \frac{1}{n-1}\sum_{i = 1}^{n} (X_{1i}^{*}-\bar{X}^{*}_{1}) \mid X_{1},X_{2},\dots,X_{n}\right) \\ & = Var(X_{11}^{*}\mid X_{1},X_{2},\dots,X_{n}) \\ & = \frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

从而

\begin{align*} E(\widehat{\text{Bias}}(\hat{\sigma}^{2})\mid X_{1},X_{2},\dots,X_{n}) & =E\left( \frac{1}{m}\sum_{i = 1}^{m} \hat{\sigma}^{2*}_{i}-\hat{\sigma}^{2} \mid X_{1},X_{2},\dots,X_{n}\right) \\ & = E(\hat{\sigma}_{1}^{2*}\mid X_{1},X_{2},\dots,X_{n})-\hat{\sigma}^{2} \\ & = \frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}- \frac{1}{n-1}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \\ & = -\frac{1}{n(n-1)}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

因此

E(\widehat{\text{Bias}}(\hat{\sigma}^{2}))=-\frac{1}{n}\sigma^{2}

1.2 估计量标准差的 Bootstrap 估计

Definition (估计量标准差的 Bootstrap 估计)

从样本 $X_{1},X_{2},\dots,X_{n}$ 中有放回地产生数据 $X^{*}_{11},X^{*}_{12},\dots,X^{*}_{1n}$ ，得到 $\theta$ 的估计并记为 $\hat{\theta}_{1}^{*}= \hat{\theta}_{1}^{*}(X^{*}_{11},X^{*}_{12},\dots,X^{*}_{1n})$
重复上述步骤 $m$ 次，相应估计分别记为 $\hat{\theta}^{*}_{1},\hat{\theta}^{*}_{2},\dots,\hat{\theta}^{*}_{m}$
计算 $\overline{\hat{\theta}}^{*}=\frac{1}{m}\sum_{i = 1}^{m} \hat{\theta}^{*}_{i}$ 此时 $\hat{\theta}$ 的标准差估计为 $\widehat{\text{SE}}(\hat{\theta})=\sqrt{ \frac{1}{m}\sum_{i = 1}^{m} (\hat{\theta}^{*}_{i}-\overline{\hat{\theta}}^{*}) }$

Example (样本均值标准差的 Bootstrap 估计)

设 $X_{1},X_{2},\dots,X_{n}$ 是来自均值为 $\mu$ ，方差为 $\sigma^{2}$ 的分布，考虑估计

\hat{\mu}=\frac{1}{n}\sum_{i = 1}^{n} X_{i}

求

E(\widehat{Var}(\hat{\mu}))

注意到

\begin{align*} Var(\hat{\mu}_{1}^{*}\mid X_{1},X_{2},\dots,X_{n}) & =Var\left( \frac{1}{n}\sum_{i = 1}^{n} X^{*}_{1i}\mid X_{1},X_{2},\dots,X_{n} \right) \\ & = \frac{1}{n}Var(X_{11}^{*}\mid X_{1},X_{2},\dots,X_{n}) \\ & = \frac{1}{n^{2}}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \end{align*}

因此

\begin{align*} E(\widehat{Var}(\hat{\mu})\mid X_{1},X_{2},\dots,X_{n}) & =E\left( \frac{1}{m}\sum_{i = 1}^{m} (\hat{\mu}^{*}_{i}- \overline{\hat{\mu}}^{*})^{2} \mid X_{1},X_{2},\dots,X_{n}\right) \\ & = \frac{m-1}{m}Var(\hat{\mu}_{1}^{*}\mid X_{1},X_{2},\dots,X_{n}) \\ & = \frac{m-1}{mn^{2}}\sum_{i = 1}^{n}(X_{i}-\bar{X})^{2} \end{align*}

从而

E(\widehat{Var}(\hat{\mu}))=\frac{(m-1)(n-1)}{mn^{2}}\sigma^{2}

1.3 基于 Bootstrap 法的置信区间估计

Definition (基于 Bootstrap 法的置信区间估计)

从样本 $(X_{1},X_{2},\dots,X_{n})$ 中有放回地产生数据 $(X_{11}^{*},X_{12}^{*},\dots,X_{1n}^{*})$ 得到的 $\theta$ 的估计，记为 $\hat{\theta}_{1}^{*}$
从所得样本 $(X_{11}^{*},X_{12}^{*},\dots,X_{1n}^{*})$ 中再抽样 $k$ 次，得到 $\hat{\theta}_{1}^{*}$ 的标准差 $\widehat{\text{SE}}(\hat{\theta}_{1}^{*})$ ，计算 $t_{1}^{*}=\frac{\hat{\theta}_{1}^{*}-\hat{\theta}}{\widehat{\text{SE}}(\hat{\theta}_{1}^{*})}$
重复上述步骤 $m$ 次，相应得到 $t_{1}^{*},t_{2}^{*},\dots,t_{m}^{*}$ ，用 $t^{*}_{\left( \left[ \frac{m\alpha}{2} \right] \right)}$ 和 $t^{*}_{\left[ \frac{m(1-\alpha)}{2} \right]}$ 表示第 $\left[ \frac{m\alpha}{2} \right]$ 和 $\left[ \frac{m(1-\alpha)}{2} \right]$ 次序统计量，则 $P\left\{ t^{*}_{\left( \left[ \frac{m\alpha}{2} \right] \right)}< \frac{\hat{\theta}-\theta}{\widehat{\text{SE}}(\hat{\theta})}< t^{*}_{\left[ \frac{m(1-\alpha)}{2} \right]}\right\}\approx 1-\alpha$ 因此置信区间为 $[\hat{\theta}-t^{*}_{\left( \left[ \frac{m\alpha}{2} \right] \right)}\widehat{\text{SE}}(\hat{\theta}),\hat{\theta}+t^{*}_{\left[ \frac{m(1-\alpha)}{2} \right]}\widehat{\text{SE}}(\hat{\theta})]$

2. Jackknife 估计

2.1 估计量偏差的 Jackknife 估计

Definition (估计量偏差的 Jackknife 估计)

\widehat{\text{Bias}}(\hat{\theta})=(n-1)\left( \frac{1}{n}\sum_{i = 1}^{n} \hat{\theta}_{(-i)}-\hat{\theta} \right)=\frac{n-1}{n}\sum_{i = 1}^{n} (\hat{\theta}_{(-i)}-\hat{\theta})

Example (样本方差偏差的 Jackknife 估计)

考虑

\hat{\theta}=\frac{1}{n}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}

计算

E(\widehat{\text{Bias}}(\hat{\theta}))

考虑

\begin{align*} E(\hat{\theta}_{(-i)}-\hat{\theta}) & =E(\hat{\theta}_{(-i)}-\theta)-E(\hat{\theta}-\theta) \\ & =\text{Bias}(\hat{\theta}_{(-i)}) - \text{Bias}(\hat{\theta}) \\ & = -\frac{\sigma^{2}}{n-1}+\frac{\sigma^{2}}{n} \\ & = - \frac{\sigma^{2}}{n(n-1)}=\frac{\text{Bias}(\hat{\theta})}{n-1} \end{align*}

因此

\begin{align*} E(\widehat{\text{Bias}}(\hat{\theta})) & =\frac{n-1}{n}\sum_{i = 1}^{n} E(\hat{\theta}_{(-i)}-\hat{\theta}) \\ & = \text{Bias}(\hat{\theta}) \end{align*}

2.2 估计量标准差的 Jackknife 估计

Definition (估计量标准差的 Jackknife 估计)

\widehat{\text{SE}}(\hat{\theta})=\sqrt{ \frac{n-1}{n}\sum_{i = 1}^{n} (\hat{\theta}_{(-i)} - \overline{\hat{\theta}}_{(\cdot)})^{2} }

Definition (样本均值标准差的 Jackknife 估计)

设总体均值为 $\theta$ ，方差为 $\sigma^{2}$ ，考虑估计

\hat{\theta}=\frac{1}{n}\sum_{i = 1}^{n} X_{i}

计算

E(\widehat{Var}(\hat{\theta}))

注意到

\begin{align*} E(\widehat{Var}(\hat{\theta})) & =E\left( \frac{n-1}{n}\sum_{i = 1}^{n}(\hat{\theta}_{(-i)}- \overline{\hat{\theta}}_{(\cdot)})^{2} \right) \\ & = (n-1)(Var(\hat{\theta}_{(-1)})-Var(\overline{\hat{\theta}}_{(\cdot)})) \\ & = (n-1)\left( \frac{\sigma^{2}}{n-1}- \frac{\sigma^{2}}{n} \right) \\ & = \frac{\sigma^{2}}{n} \end{align*}

2.3 Jackknife-after-Bootstrap 估计

Definition (Jackknife-after-Bootstrap 估计)

固定 $i$ ，从观测数据 $X_{(-i)}=(X_{1},\dots,X_{i-1},X_{i+1},\dots, X_{n})$ 中有放回地产生数据 $X^{*}_{(-i),1}=(X_{11}^{*},X_{12}^{*},\dots,X_{1,n-1}^{*})$ ，基于此样本得到 $\theta$ 的估计为 $\hat{\theta}_{(-i),1}^{*}$
重复上述步骤 $m$ 次，相应估计分别记为 $\hat{\theta}^{*}_{(-i),1},\hat{\theta}^{*}_{(-i),2},\dots,\hat{\theta}^{*}_{(-i),m}$
计算 $\overline{\hat{\theta}}^{*}_{(-i)}=\frac{1}{m}\sum_{k = 1}^{m}\hat{\theta}^{*}_{(-i),k}$ 。基于 $\hat{\theta}^{*}_{(-i),1},\hat{\theta}^{*}_{(-i),2},\dots,\hat{\theta}^{*}_{(-i),m}$ ，得到 $\hat{\theta}$ 的标准差的 Bootstrap 估计为 $\widehat{\text{SE}}_{(-i)}(\hat{\theta})=\sqrt{ \frac{1}{m}\sum_{k = 1}^{m} (\hat{\theta}_{(-i),k}^{*}-\overline{\hat{\theta}}^{*}_{(-i)})^{2} }$
对于 $i=1,2,\dots,n$ ，重复上述步骤，得到 $\widehat{\text{SE}}_{(-1)}(\hat{\theta}),\widehat{\text{SE}}_{(-2)}(\hat{\theta}),\dots,\widehat{\text{SE}}_{(-n)}(\hat{\theta})$ ，则 $\widehat{\text{SE}}(\hat{\theta})$ 的标准差的 Jackknife 估计为 $\widehat{\text{SE}}_{Jack}(\widehat{\text{SE}}(\hat{\theta}))=\sqrt{ \frac{n-1}{n}\sum_{i = 1}^{n} (\widehat{\text{SE}}_{(-i)}(\hat{\theta})- \overline{\widehat{\text{SE}}}_{(\cdot)}(\hat{\theta}) )^{2} }$

06重抽样算法

ON THIS PAGE