06广义线性回归模型

Lingfeng 2024-12-10

06广义线性回归模型

1. 模型形式

1.1 指数分布族

Definition (指数分布族)

f(y;\theta,\phi)=\exp \left\{ \frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi) \right\}

Theorem (指数分布族性质)

分布均值为 $E(Y)=b'(\theta)$ 分布方差为 $Var(Y)=a(\phi)b''(\theta)$ $b''(\theta)$ 可以表示为 $\mu$ 的函数，即方差函数 $V(\mu)$ 。

证明
注意到 $\int_{-\infty}^{\infty} f(y;\theta,\phi) \, dy=1$ 此时 $\begin{align*} \frac{d}{d\theta} \int_{-\infty}^{\infty} f(y;\theta,\phi) \, dy & = \int_{-\infty}^{\infty} \frac{df(y;\theta,\phi)}{d\theta} \, dy \\ & = \int_{-\infty}^{\infty} \frac{y-b'(\theta)}{a(\phi)}f(y;\theta,\phi) \, dx \\ & = 0 \end{align*}$ 移项即得 $E(y)=b'(\theta)$ 同时两边再求导有 $\int_{-\infty}^{\infty} \left( -\frac{b''(\theta)}{a(\phi)}f(y;\theta,\phi) \right)+\left( \frac{y-b'(\theta)}{a(\phi)} \right)^{2}f(y;\theta,\phi) \, dy =0$ 化简即得 $Var(Y)=a(\phi)b''(\theta)$

1.2 常见指数分布族

分布	$a(\phi)$	$\theta$	$b(\theta)$	$c(y,\theta)$
正态	$\sigma^{2}$	$\mu$	$\frac{\theta^{2}}{2}$	$-\frac{1}{2}\left( \frac{y^{2}}{\phi}+\log(2\pi \phi) \right)$
二项	1	$\log\left( \frac{p}{1-p} \right)$	$m\log(1+e^{\theta})$	$\log C^{y}_{m}$
泊松	1	$\log \mu$	$e^{\theta}$	$-\log(y!)$

证明
正态分布 $\begin{align*} f(y) & =\frac{1}{\sqrt{ 2\pi }\sigma}\exp \left\{ -\frac{(y-\mu)^{2}}{2\sigma^{2}} \right\} \\ & = \exp \left\{ \frac{y\mu-\mu^{2} / 2}{\sigma^{2}} -\frac{1}{2}\left( \frac{y}{\sigma^{2}}+\log(2\pi\sigma^{2}) \right) \right\} \\ & =\exp \left\{ \frac{y\theta-\theta^{2} / 2}{\phi}-\frac{1}{2}\left( \frac{y}{\phi}+\log(2\pi \phi) \right) \right\} \end{align*}$ 二项分布 $\begin{align*} f(y) & =C_{m}^{y}p^{y}(1-p)^{m-y} \\ & = \exp \left\{ y\ln\left( \frac{p}{1-p} \right)+m\ln(1-p)+\log(C^{y}_{m}) \right\} \\ & = \exp \{ y\theta-m\log(1+e^{\theta})+\log(C^{y}_{m}) \} \end{align*}$ 泊松分布 $\begin{align*} f(y) & =\frac{\lambda^{y}}{y!}e^{-\lambda} \\ & = \exp \{ y\ln\lambda-\lambda-\ln(y!) \} \\ & =\exp \{ y\theta-e^{\theta}-\ln(y!) \} \end{align*}$

1.3 模型构成

Definition (广义线性回归模型构成)

广义线性回归模型由三部分构成：
随机成分 $Y_{i}\sim f(y_{i};\theta_{i},\phi)$ 系统成分 $\eta_{i}=X_{i}\beta=\sum_{j = 0}^{p} \beta_{j}X_{ij}$ 连接函数 $\eta_{i}=g(\mu_{i})$ 其中使得 $\eta_{i}=\theta_{i}$ 成立的 $g(\cdot)$ 称为典型连接函数。

2. 二分类Logistic回归模型

2.1 定义

Definition (Logistic回归)

响应变量 $Y$ 的概率密度函数为 $f(y;p)=p^{y}(1-p)^{1-y}$ 连接函数为 $\mathrm{logit}(p_{i})=\log\left( \frac{p_{i}}{1-p_{i}} \right)=X_{i}\beta$

2.2 模型求解

考虑极大似然法，对数似然函数为 $l(\beta)=\sum_{i = 1}^{n} \left( \frac{y_{i}\theta_{i}-b(\theta_{i})}{a(\phi)}+c(y_{i},\phi) \right)$ 考虑 $\begin{align*} \frac{ \partial l(\beta) }{ \partial \beta_{j} } & =\sum_{i = 1}^{n} \frac{ \partial l_{i}(\beta) }{ \partial \beta_{j} } \\ & = \sum_{i = 1}^{n} \frac{ \partial l_{i}(\beta) }{ \partial \theta_{i} } \frac{ \partial \theta_{i} }{ \partial \mu_{i} } \frac{ \partial \mu_{i} }{ \partial \eta_{i} } \frac{ \partial \eta_{i} }{ \partial \beta_{j} } \\ & = \sum_{i = 1}^{n} \left( \frac{y_{i}-\mu_{i}}{a(\phi)} \right)(V(\mu_{i}))^{-1}(g'(\mu_{i}))^{-1}X_{ij} \\ \end{align*}$ 此时取 $z_{i}=\eta_{i}+g'(\mu_{i})(y_{i}-\mu_{i})$ ，此时 $U_{i}(\beta)=\sum_{i = 1}^{n} \omega_{i}X_{ij}(z_{i}-\eta_{i})$ 其中 $\omega_{i}=(V(\mu_{i})a(\phi)(g'(\mu_{i}))^{2})^{-1}$ 。

其实本质上是构造 $g(y_{i})$ 一阶泰勒展开。注意到 $g(y_{i})=g(\mu_{i})+g'(\mu_{i})(y_{i}-\mu_{i})$ 因此 $Var(z_{i})=Var(y_{i})(g'(\mu_{i}))^{2}=V(\mu_{i})a(\phi)(g'(\mu_{i})^{2})$

2.3 偏差

Definition (偏差)

\begin{align*}D(y,\hat{\mu})&=2a(\phi)(l(\tilde{\theta}-l(\hat{\theta}))) \\ &= 2\sum_{i = 1}^{n} (y_{i}(\tilde{\theta}_{i}-\hat{\theta}_{i})-(b(\tilde{\theta}_{i})-b(\hat{\theta}_{i})))\end{align*}

若相应变量服从正态分布，则 $b(\theta_{i})=\frac{\theta_{i}^{2}}{2}$ 且 $\theta_{i}=\mu_{i}$ ，此时 $\begin{align*} D(y,\hat{\mu}) & =2\sum_{i = 1}^{n} \left( y_{i}(y_{i}-\hat{\mu}_{i})-\frac{y_{i}^{2}}{2}+\frac{\hat{\mu}_{i}^{2}}{2} \right) \\ & = \sum_{i = 1}^{n} (y_{i}-\hat{\mu}_{i})^{2} \end{align*}$
若服从伯努利分布，则 $\theta_{i}=\log \frac{p_{i}}{1-p_{i}}$ 且 $b(\theta_{i})=\log(1+e^{\theta_{i}})=\log \frac{1}{1-p_{i}}$ ，此时 $\begin{align*} D(y,\hat{p}) & =2\sum_{i = 1}^{n} (y_{i}(\tilde{\theta}-\hat{\theta})-(b(\tilde{\theta}_{i})-b(\hat{\theta}))) \\ & = 2\sum_{i = 1}^{n} \left( y_{i}\log \frac{y_{i}}{\hat{p}_{i}}+(1-y_{i})\left( \log \frac{1-y_{i}}{1-p_{i}} \right) \right) \\ & =-2\sum_{i = 1}^{n} (y_{i}\log \hat{p}_{i}+(1-y_{i})\log(1-\hat{p}_{i})) \end{align*}$

3. 多分类Logistic回归模型

Definition (多分类Logistic回归)

\begin{cases} P\{ Y_{i}=k \} = \frac{1}{1+\sum_{j = 2}^{K}\exp \{ X_{i}\beta_{i} \} }, & k=1 \\ P\{ Y_{i} =k \}=\frac{\exp \{ X_{i}\beta_{i} \}}{1+\sum_{j = 2}^{K}\exp \{ X_{i}\beta_{i} \} }, & k=2,..,K\end{cases}

4. Possion回归模型

Definition (Possion回归)

概率密度函数为 $f(y;\mu)=\frac{\mu^{y}e^{-\mu}}{y!}$ 连接函数 $\log \mu_{i}=X_{i}\beta$