01一元线性回归

Lingfeng 2024-10-08

01一元线性回归

1. 定义

Definition (一元线性回归模型)

给定样本 $(X_{1},Y_{1})$ ， $(X_{2},Y_{2})$ ， $\dots$ ， $(X_{n},Y_{n})$ ，有样本回归模型 $Y_{i}=\beta_{0}+\beta_{1}X_{i}+\varepsilon_{i},\quad i=1,\dots,n$ 其中 $Y_{1},Y_{2},\dots,Y_{n}$ 以及 $\varepsilon_{1},\varepsilon_{2},\dots,\varepsilon_{n}$ 都是相互独立的随机变量，而 $X_{i}$ 是非随机变量。

其中随机误差项 $\varepsilon$ 满足高斯-马尔可夫条件（Gauss-Markov）条件，即 $\begin{cases} E(\varepsilon_{i})=0, & i=1,\dots,n \\ Var(\varepsilon_{i})=\sigma^{2}, & i=1,\dots,n \\ Cov(\varepsilon_{i},\varepsilon_{j}) =0, & i\neq j \end{cases}$ 同时满足正态性假定 $\varepsilon_{i}\sim N(0,\sigma^{2}),\quad i=1,\dots,n$

Corollary ( $Y$ 的性质)

在高斯-马尔科夫条件下，有 $\begin{align*} E(Y_{i})&=\beta_{0} +\beta_{1}X_{i} \\ Var(Y_{i})&=\sigma^{2} \end{align*}$ 在正态性假设下有 $Y_{i}\sim N(\beta_{0}+\beta_{1}X_{i},\sigma^{2})$

2. 参数估计方法

Theorem ( $\hat{\beta_{0}}$ 和 $\hat{\beta_{1}}$ 表达式)

\begin{align*} \hat{\beta_{1}}&= \frac{L_{xy}}{L_{xx}} \\ \hat{\beta_{0}}&= \bar{Y}-\hat{\beta_{1}}\bar{X}\end{align*}

2.1 最小二乘法

考虑 $\begin{align*} Q(\hat{\beta_{0}},\hat{\beta_{1}}) & =\sum_{i = 1}^{n} (Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i} )^{2} \\ & = \min_{\beta_{0},\beta_{1}}\sum_{i = 1}^{n} (Y_{i}-\beta_{0}-\beta_{1}X_{i})^{2} \end{align*}$
此时考虑 $\begin{align*} \frac{ \partial Q }{ \partial \beta_{0} }\Big|_{\beta_{0}=\hat{\beta_{0}}} & =-2\sum_{i = 1}^{n} (Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i})=0 \\ \frac{ \partial Q }{ \partial \beta_{1} } \Big|_{\beta_{1}=\hat{\beta_{1}}} & =-2\sum_{i = 1}^{n} (Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i})X_{i}=0 \end{align*}$ 解得 $\begin{align*} \hat{\beta_{0}} & =\bar{Y}-\hat{\beta_{1}}\bar{X} \\ \hat{\beta_{1} } & = \frac{\sum_{i = 1}^{n} (X_{i}-\bar{X})(Y_{i}-\bar{Y})}{\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}}= \frac{L_{xy}}{L_{xx}} \end{align*}$

2.2 极大似然法

注意到 $Y_{i}\sim N(\beta_{0}+\beta_{1}X_{i},\sigma^{2})$ 因此构建 $L(\beta_{0},\beta_{1},\sigma^{2})=\prod_{i=1}^{n} \frac{1}{\sqrt{ 2\pi }\sigma}\exp \left\{ -\frac{(Y_{i}-\beta_{0}-\beta_{1}X_{i})^{2}}{2\sigma^{2}} \right\}$ 此时 $l(\beta_{0},\beta_{1},\sigma^{2})=-\frac{n}{2}\ln\sigma^{2}-\frac{\sum_{i = 1}^{n} (Y_{i}-\beta_{0}-\beta_{1}X_{i})^{2}}{2\sigma^{2}}$ 同样考虑 $\begin{align*} \frac{ \partial l }{ \partial \sigma^{2} } & = - \frac{n}{2\sigma^{2}}+ \frac{\sum_{i = 1}^{n} (Y_{i}-\beta_{0}-\beta_{1}X_{i})^{2}}{2\sigma^{4}}=0 \\ \frac{ \partial l }{ \partial \beta_{0} } & = \frac{\sum_{i = 1}^{n} (Y_{i}-\beta_{0}-\beta_{1}X_{i})}{\sigma^{2}}=0\\ \frac{ \partial l }{ \partial \beta_{1} } & = \frac{\sum_{i = 1}^{n} (Y_{i}-\beta_{0}-\beta_{1}X_{i})X_{i}}{\sigma^{2}}=0 \end{align*}$ 此时得到相同结果，并得到方差的极大似然估计 $\hat{\sigma^{2}}= \frac{1}{n}\sum_{i = 1}^{n} (Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i})^{2}= \frac{1}{n}\sum_{i = 1}^{n} (Y_{i}-\hat{Y_{i}})^{2}$

2.3 矩估计

对于随机变量 $\varepsilon_{i}$ ，有 $\begin{align*} E(\varepsilon_{i}) & =0 \\ E(X_{i}\varepsilon_{i}) & = 0 \end{align*}$ 使用矩估计 $\begin{align*} \hat{\varepsilon_{i}} & = \frac{1}{n} (Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i})=0 \\ X_{i}\hat{\varepsilon_{i}} & = \frac{1}{n}X_{i}(Y_{i}-\hat{\beta_{0}}-\hat{\beta_{1}}X_{i})=0 \end{align*}$ 显然也能得到相同条件。

3. 参数估计性质

3.1 线性

Theorem (线性)

估计量 $\hat{\beta_{0}}$ ， $\hat{\beta_{1}}$ 为随机变量 $Y_{i}$ 的线性函数。其中 $\begin{align*} \hat{\beta_{1}} & = \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X} )}{L_{xx}}Y_{i} \\ \hat{\beta_{0}} & = \sum_{i = 1}^{n} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{xx}} \right) Y_{i}\end{align*}$

证明
注意到 $\begin{align*} \hat{\beta_{1}} & = \frac{L_{xy}}{L_{x x}} \\ & = \frac{\sum_{i = 1}^{n} (X_{i}-\bar{X})(Y_{i}-\bar{Y})}{\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}} \\ & = \frac{\sum_{i = 1}^{n} (X_{i}-\bar{X})Y_{i}}{\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}} \\ & = \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2 }} Y_{i} \\ & = \sum_{i = 1}^{n} \frac{ (X_{i}-\bar{X})}{L_{x x}}Y_{i} \end{align*}$ 因此 $\begin{align*} \hat{\beta_{0}} & = \bar{Y}-\hat{\beta_{1}}\bar{X} \\ & = \frac{1}{n} \sum_{i = 1}^{n}Y_{i} - \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})\bar{X}}{L_{x x}} Y_{i} \\ & = \sum_{i = 1}^{n} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{x x}} \right)Y_{i} \end{align*}$

3.2 无偏性

Theorem (无偏性)

$\hat{\beta_{0}}$ ， $\hat{\beta_{1}}$ 是 $\beta_{0}$ ， $\beta_{1}$ 的无偏估计。即 $\begin{align*} E(\hat{\beta_{1}})&=\beta_{1} \\ E(\hat{\beta_{0}})&= \beta_{0}\end{align*}$

证明
考虑 $E(Y_{i})=\beta_{0}+\beta_{1}X_{i}$ 因此 $\begin{align*} E(\hat{\beta_{1}}) & = \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{L_{x x}}E(Y_{i}) \\ & = \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{L_{x x}}(\beta_{0}+\beta_{1}X_{i}) \\ & = \beta_{1}\sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})X_{i}}{L_{x x}} \\ & = \beta_{1} \end{align*}$ 同时 $E(\bar{Y})=\beta_{0}+\beta_{1}\bar{X}$ 因此 $E(\hat{\beta_{0}})=E(\bar{Y}-\hat{\beta_{1}}\bar{X})=\beta_{0}$

3.3 $\hat{\beta_{0}}$ 和 $\hat{\beta_{1}}$ 方差

Theorem ( $\hat{\beta_{0}}$ 和 $\hat{\beta_{1}}$ 方差)

\begin{align*} Var(\hat{\beta_{1}}) & = \frac{\sigma^{2}}{L_{xx}} \\ Var(\hat{\beta_{2}})&= \left( \frac{1}{n} + \frac{\bar{X}^{2}}{L_{x x}} \right)\sigma^{2 }=\frac{\sum_{i = 1}^{n} X_{i}^{2}}{nL_{xx}}\sigma^{2} \end{align*}

证明
由于 $Y_{i}$ 相互独立，因此 $\begin{align*} Var(\hat{\beta_{1}}) & =\sum_{i = 1}^{n} \left( \frac{(X_{i}-\bar{X})}{L_{x x} } \right)^{2}Var(Y_{i}) \\ & = \frac{\sigma^{2}}{L_{x x}} \end{align*}$ 同理 $\begin{align*} Var(\hat{\beta_{0}}) & =\sum_{i = 1}^{n} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{x x}} \right)^{2} \sigma^{2}\\ & = \left( \frac{1}{n} + \frac{\bar{X}^{2}}{L_{x x}} \right)\sigma^{2 } \\ & = \frac{\sum_{i = 1}^{n} X_{i}^{2}}{nL_{xx}}\sigma^{2} \end{align*}$

Corollary ( $\hat{\beta_{0}}$ 和 $\hat{\beta_{1}}$ 的分布)

\begin{align*} \hat{\beta_{0}} & \sim N\left( \beta_{0}, \frac{\sum_{i = 1}^{n} X_{i}^{2}}{nL_{xx}}\sigma^{2} \right) \\ \hat{\beta_{1}} & \sim N\left( \beta_{1}, \frac{\sigma^{2}}{L_{x x}} \right)\end{align*}

3.4 $\hat{\beta_{0}}$ 与 $\hat{\beta_{1}}$ 的协方差

\begin{align*} Cov(\hat{\beta_{0}},\hat{\beta_{1}} ) & = Cov\left( \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{L_{x x}}Y_{i}, \sum_{i = 1}^{n} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{x x} } \right) Y_{i}\right) \\ & = \sigma^{2}\sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{L_{xx}} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{xx}} \right) \\ & = -\frac{\bar{X}}{L_{x x}} \sigma^{2} \end{align*}

Corollary ( $\hat{\beta_{0}}$ 与 $\hat{\beta_{1}}$ 相关性)

当 $\bar{X}=0$ 时， $\hat{\beta_{0}}$ 与 $\hat{\beta_{1}}$ 相互独立。

3.5 高斯-马尔科夫定理

Theorem (高斯-马尔科夫定理)

在满足高斯-马尔可夫条件下， $\hat{\beta_{0}}$ 和 $\hat{\beta_{1}}$ 是 $\beta_{0}$ 和 $\beta_{1}$ 的最佳无偏估计，即最小方差线性无偏估计(BLUE)。

证明
设 $\tilde{\beta_{1}}=\sum_{i = 1}^{n}a_{i}Y_{i}$ ，且其为 $\beta_{1}$ 的无偏估计。此时有 $\begin{align*} E(\tilde{\beta_{1}}) & =\sum_{i = 1}^{n} a_{i}(\beta_{0}+\beta_{1}X_{i}) \\ & = \beta_{0}\sum_{i = 1}^{n} a_{i}+\beta_{1}\sum_{i = 1}^{n} a_{i}X_{i} \\ & = \beta_{1} \end{align*}$ 因此得到 $\begin{align*} \sum_{i = 1}^{n} a_{i} & =0 \\ \sum_{i = 1}^{n} a_{i}X_{i} & = 1 \end{align*}$ 此时 $\begin{align*} Var(\tilde{\beta_{1}}) & = Var(\hat{\beta_{1}}+ \tilde{\beta_{1}}-\hat{\beta_{1}}) \\ & = Var(\hat{\beta_{1}})+Var(\tilde{\beta_{1}}-\hat{\beta_{1}})+2Cov(\hat{\beta_{1}}, \tilde{\beta_{1}}-\hat{\beta_{1}}) \end{align*}$ 注意到 $\begin{align*} Cov(\hat{\beta_{1}}, \tilde{\beta_{1}}-\hat{\beta_{1}}) & = Cov(\hat{\beta_{1}}, \tilde{\beta_{1}})-Var(\hat{\beta_{1}}) \\ & = Cov\left( \sum_{i = 1}^{n} \frac{(X_{i}-\bar{X})}{L_{x x}}Y_{i}, \sum_{i = 1}^{n} a_{i}Y_{i} \right)- \frac{\sigma^{2}}{L_{x x}} \\ & = \frac{\sigma^{2}}{L_{x x}} \sum_{i = 1}^{n} a_{i} (X_{i}-\bar{X})- \frac{\sigma^{2}}{L_{x x}} \\ & = 0 \end{align*}$ 故证毕。

同理，设 $\tilde{\beta_{0}}=\sum_{i = 1}^{n}b_{i}Y_{i}$ ，也有 $\begin{align*} \sum_{i = 1}^{n} b_{i} & =1 \\ \sum_{i = 1}^{n} b_{i}X_{i} & =0 \end{align*}$ 此时 $\begin{align*} Cov(\hat{\beta_{0}}, \tilde{\beta_{0}}-\hat{\beta_{0}}) & = Cov(\hat{\beta_{0}}, \tilde{\beta_{0}})-Var(\hat{\beta_{0}}) \\ & = Cov\left( \sum_{i = 1}^{n} \left( \frac{1}{n}- \frac{(X_{i}-\bar{X})\bar{X}}{L_{xx}} \right)Y_{i}, \sum_{i = 1}^{n} b_{i}Y_{i} \right)- \left( \frac{1}{n}+ \frac{\bar{X}^{2}}{L_{xx}} \right)\sigma^{2} \\ & = \sigma^{2}\sum_{i = 1}^{n} b_{i}\left( \frac{1}{n}- \frac{(X_{i}-\bar{X} )\bar{X}}{L_{xx}} \right)- \left( \frac{1}{n}+ \frac{\bar{X}^{2}}{L_{xx}} \right)\sigma^{2} \\ & =0 \end{align*}$ 故证毕。

4. 模型推断

4.1 t检验

我们建立统计假设为 $\begin{align*} H_{0}: \beta_{1}=0 \\ H_{1}:\beta_{1}\neq 0 \end{align*}$
此时若原假设成立，有 $\hat{\beta_{1}}\sim N\left( 0, \frac{\sigma^{2}}{L_{xx}} \right)$ 因此构造 $t= \frac{\hat{\beta_{1}}}{\hat{\sigma} /\sqrt{ L_{xx} }}= \frac{\hat{\beta_{1}}\sqrt{ L_{xx} }}{\hat{\sigma}}$ 其中 $\hat{\sigma}^{2}= \frac{1}{n-2}\sum_{i = 1}^{n} (Y_{i}-\hat{Y})^{2}$ 此时 $t$ 自由度为 $n-2$ 的 $t$ 分布。

4.2 F检验

Theorem (平方和分解式)

$SST=SSR+SSE$ 即 $\sum_{i = 1}^{n} (Y_{i}-\bar{Y_{i}})^{2}=\sum_{i = 1}^{n} (\hat{Y_{i}}-\bar{Y})^{2}+\sum_{i = 1}^{n} (Y_{i}-\hat{Y_{i}})^{2}$

此时构造 $F= \frac{SSR / 1}{SSE / (n-2)}$ 服从自由度为 $(1,n-2)$ 的 $F$ 分布。

Theorem (t检验和F检验的等价性)

t^{2}=F

证明
注意到 $\begin{align*} SSR & = \sum_{i = 1}^{n} (\hat{Y_i}-\bar{Y})^{2} \\ & = \hat{\beta_{1}}^{2}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \\ & = \hat{\beta_{1}}^{2}L_{xx} \end{align*}$ 代入即得结果。

4.3 相关系数

Definition (相关系数)

r= \frac{\sum_{i = 1}^{n}(X_{i}-\bar{X}) (Y_{i}-\bar{Y})}{\sqrt{ \sum_{i = 1}^{n} (X_{i}-\bar{X})^{2}\sum_{i = 1}^{n} (Y_{i}-\bar{Y})^{2} }}= \frac{L_{xy}}{\sqrt{ L_{xx}L_{yy} }}=\hat{\beta_{1}}\sqrt{ \frac{L_{xx}}{L_{yy}} }

Definition (决定系数)

R^{2}= \frac{SSR}{SST}=\frac{\sum_{i = 1}^{n} (\hat{Y_i}-\bar{Y})^{2}}{\sum_{i = 1}^{n} (Y_{i}-\bar{Y})^{2}}

Theorem (相关系数与决定系数的关系)

r^{2}=R^{2}

证明
显然 $\begin{align*} R^{2} & = \frac{SSR}{SST} \\ & = \frac{\hat{\beta_{1}}^{2}L_{xx}}{L_{xy}} \\ & = r^{2} \end{align*}$

Theorem (决定系数与t检验的关系)

t= \frac{\sqrt{ n-2 }r}{\sqrt{ 1-r^{2} }}

证明
只需证明 $\frac{(n-2)r^{2}}{1-r^{2}}=F= \frac{SSR / 1}{SSE / (n-2)}$ 即证 $\frac{r^{2}}{1-r^{2}}= \frac{SSR}{SSE}$ 而 $r^{2}=R^{2}=\frac{SSR}{SST}$ 故显然成立。

5. 残差分析

Theorem (性质一)

E(e_{i})=0

Theorem (性质二)

$\begin{align*} Var(e_{i})&=(1-h_{ii})\sigma^{2} \\ Cov(e_{i},e_{j}) & = -h_{ij}\sigma^{2}\end{align*}$ 其中 $h_{ij}=\frac{1}{n}+\frac{(X_{i}-\bar{X})(X_{j}-\bar{X})}{L_{xx}}$

证明
注意到 $\begin{align*} Var(\hat{Y_i}) & =Var(\hat{\beta_{1}}X_{i}+\hat{\beta_{0}} ) \\ & = X_{i}^{2 }Var(\hat{\beta_{1}} )+Var(\hat{\beta_{0}} ) +2X_{i}Cov(\hat{\beta_{1}} ,\hat{\beta_{0}} )\\ & = \frac{X_{i}^{2}}{L_{xx}}\sigma^{2}+\left( \frac{1}{n}+\frac{\bar{X}^{2}}{L_{xx}} \right)\sigma^{2}-\frac{2X_{i}\bar{X}}{L_{xx}}\sigma^{2} \\ & = \left( \frac{1}{n}+ \frac{(X_{i}-\bar{X})^{2}}{L_{xx}} \right)\sigma^{2} \\ & = h_{ii}\sigma^{2} \end{align*}$ 同时 $\begin{align*} \hat{Y_i} & = \hat{\beta_{1}} X_{i}+\hat{\beta_{0}} \\ & = \sum_{k = 1}^{n} \frac{(X_{k}-\bar{X})X_{i}}{L_{xx}}Y_{k}+\sum_{k = 1}^{n} \left( \frac{1}{n}- \frac{(X_{k}-\bar{X})\bar{X}}{L_{xx}} \right)Y_{k} \\ & = \sum_{k = 1}^{n} \left( \frac{1}{n}+ \frac{(X_{k}-\bar{X})(X_{i}-\bar{X})}{L_{xx}} \right)Y_{k} \end{align*}$ 因此 $Cov(\hat{Y_i},Y_{i})=\left( \frac{1}{n} + \frac{(X_{i}-\bar{X})^{2}}{L_{xx}} \right)\sigma^{2}=h_{ii}\sigma^{2}$ 此时 $\begin{align*} Var(e_{i}) & =Var(Y_{i}-\hat{Y_i}) \\ & = Var(Y_{i})+Var(\hat{Y_i})-2Cov(Y_{i},\hat{Y_i}) \\ & = \sigma^{2}+ h_{ii}\sigma^{2}-2h_{ii}\sigma^{2} \\ & =(1-h_{ii})\sigma^{2} \end{align*}$

下面计算协方差。注意到 $\begin{align*} Cov(e_{i},e_{j}) & =Cov(Y_{i}-\hat{Y_i},Y_{j}-\hat{Y_j}) \\ & = Cov(\hat{Y_i},\hat{Y_j})-Cov(Y_{i},\hat{Y_j})-Cov(\hat{Y_i},Y_{j}) \end{align*}$ 此时 $\begin{align*} Cov(\hat{Y_i},\hat{Y_j}) & = Cov\left( \sum_{k = 1}^{n} \left( \frac{1}{n}+ \frac{(X_{k}-\bar{X})(X_{i}-\bar{X})}{L_{xx}} \right) Y_{k}, \sum_{k = 1}^{n} \left( \frac{1}{n}+ \frac{(X_{k}-\bar{X})(X_{j}-\bar{X})}{L_{xx}} \right)Y_{k}\right) \\ & = \sigma^{2}\sum_{k = 1}^{n} \left( \frac{1}{n}+ \frac{(X_{k}-\bar{X})(X_{i}-\bar{X})}{L_{xx}} \right)\left( \frac{1}{n}+\frac{(X_{k}-\bar{X})(X_{j}-\bar{X})}{L_{xx}} \right) \\ & = \sigma^{2}\left( \frac{1}{n}+ \frac{(X_{i}-\bar{X})(X_{j}-\bar{X})}{L_{xx}} \right) \\ & = h_{ij}\sigma^{2} \end{align*}$ 同时 $Cov(Y_{i},\hat{Y_j}) = \left( \frac{1}{n}+ \frac{(X_{i}-\bar{X})(X_{j}-\bar{X})}{L_{xx}} \right)\sigma^{2} = h_{ij}\sigma^{2}$ 因此 $\begin{align*} Cov(e_{i},e_{j}) & = h_{ij}\sigma^{2}-2h_{ij}\sigma^{2}=-h_{ij}\sigma^{2} \end{align*}$

Corollary ( $\hat{Y_i}$ 的分布)

\hat{Y_i}\sim N\left( \beta_{1}X_{i}+\beta_{0},\left( \frac{1}{n}+ \frac{(X_{i}-\bar{X})^{2}}{L_{xx}} \right)\sigma^{2} \right)=N(Y_{i},h_{ii}\sigma^{2})

Corollary ( $\sigma^{2}$ 的无偏估计)

\hat{\sigma^{2}}=\frac{1}{n-2}\sum_{i = 1}^{n} (Y_{i}-\hat{Y_i})^{2}= \frac{1}{n-2}\sum_{i = 1}^{n} e_{i}^{2}

证明

法一
注意到 $\begin{align*} E(SST) & = E\left( \sum_{i = 1}^{n} (Y_{i}-\bar{Y})^{2} \right) \\ & = E\left( \sum_{i = 1}^{n} (\beta_{1}(X_{i}-\bar{X})+(\varepsilon_{i}-\bar{\varepsilon}))^{2} \right) \\ & = \beta_{1}^{2}L_{xx}+ E\left( \sum_{i = 1}^{n} (\varepsilon_{i}-\bar{\varepsilon})^{2} \right) \\ & = \beta_{1}^{2}L_{xx}+(n-1)\sigma^{2} \end{align*}$ 同时 $\begin{align*} SSR & = \sum_{i = 1}^{n} (\hat{Y_i}-\bar{Y})^{2} \\ & = \hat{\beta_{1}^{2}}\sum_{i = 1}^{n} (X_{i}-\bar{X})^{2} \\ & = \hat{\beta_{1}^{2}}L_{xx} \end{align*}$ 注意到 $\begin{align*} E(\hat{\beta_{1}^{2}} ) & =Var(\hat{\beta_{1}} )+E^{2}(\hat{\beta_{1}} ) \\ & = \frac{\sigma^{2}}{L_{xx}}+ \beta_{1}^{2} \end{align*}$ 因此 $\begin{align*} E(SSR) & =E(\hat{\beta_{1}^{2}} )L_{xx} \\ & = \left( \frac{\sigma^{2}}{L_{xx}}+ \beta_{1}^{2} \right) L_{xx} \\ & =\sigma^{2 }+\beta_{1}^{2}L_{xx} \end{align*}$ 最后 $\begin{align*} E(SSE) & =E(SST)-E(SSR) \\ & = (n-2)\sigma^{2} \end{align*}$

法二
$\begin{align*} E\left( \sum_{i = 1}^{n} e_{i}^{2} \right) & = \sum_{i = 1}^{n} E(e_{i}^{2}) \\ & = \sum_{i = 1}^{n} Var(e_{i}) \\ & = \sum_{i = 1}^{n} \left( 1- \frac{1}{n}- \frac{(X_{i}-\bar{X})^{2}}{L_{xx}} \right)\sigma^{2} \\ & =(n-2)\sigma^{2} \end{align*}$

Theorem (性质三)

\begin{align*} \sum_{i = 1}^{n} e_{i} &=0 \\ \sum_{i = 1}^{n} X_{i}e_{i}&=0\end{align*}

Definition (改进残差)

\begin{align*} ZRE_{i}&= \frac{e_{i}}{\hat{\sigma}} \\ SRE_{i}&= \frac{e_{i}}{\hat{\sigma}\sqrt{ 1-h_{i i} }}\end{align*}

6. 回归系数的区间估计

Theorem ( $\beta_{1}$ 的置信区间)

$\beta_{1}$ 的置信度为 $1-\alpha$ 的置信区间为 $\left( \hat{\beta_{1}} -t_{\frac{\alpha}{2}} \frac{\hat{\sigma}}{\sqrt{ L_{xx} }},\hat{\beta_{1}}+t_{\frac{\alpha}{2}} \frac{\hat{\alpha}}{\sqrt{ L_{xx} }} \right)$

证明
注意到 $t=\frac{(\beta_{1}-\hat{\beta_{1}} )\sqrt{ L_{xx} }}{\hat{\sigma}}=t(n-2)$ 即得 $P\left\{ \left\lvert \frac{(\beta_{1}-\hat{\beta_{1}})\sqrt{ L_{xx} }}{\hat{\sigma}} \right\rvert <t_{\frac{\alpha}{2}} \right\}=1-\alpha$

7. 模型预测

7.1 单值预测

给定一个新样本 $X_{0}$ ，其对应预测值为 $\hat{Y_0}=\hat{\beta_{1}}X_{0}+\hat{\beta_{0}}$ 其为目标值 $Y_{0}$ 的无偏估计。

7.2 因变量新值的区间估计

Theorem ( $\hat{Y_0}$ 的分布)

\hat{Y_0}\sim N\left( \beta_{1}X_{0}+\beta_{0},\left( \frac{1}{n}+ \frac{(X_{0}-\bar{X})^{2}}{L_{xx}}\sigma^{2} \right) \right)=N(Y_{0},h_{00}\sigma^{2})

证明
由之前定理知显然成立。

因此 $Y_{0}-\hat{Y_0}\sim N(0,(1+h_{00})\sigma^{2})$ 此时得到统计量 $t=\frac{Y_{0}-\hat{Y_0}}{\sqrt{ 1+h_{00} }\hat{\sigma}}\sim t(n-2)$ 得到置信度为 $1-\alpha$ 置信区间为 $\hat{Y_0}\pm t_{\frac{\alpha}{2}}(n-2)\sqrt{ 1+h_{00} }\hat{\sigma}$ 当样本量较大， $\lvert X_{i}-\bar{X} \rvert$ 较小时， $h_{00}$ 接近于零，因此置信度 $95\%$ 的置信区间近似为 $\hat{Y_0}\pm 2\hat{\sigma}$

7.3 因变量新值的平均值的区间预测

注意到 $E(Y_{0})$ 为常数，因此 $\hat{Y_0}-E(Y_{0})\sim (0,h_{00}\sigma^{2})$ 因此置信度为 $1-\alpha$ 置信区间为 $\hat{Y_0}\pm t_{\frac{\alpha}{2}}(n-2)\sqrt{ h_{00} }\hat{\sigma}$