01 A Brief Review of Linear Regression

Lingfeng 2026-09-15

01 A Brief Review of Linear Regression

1. Model

Definition (Multiple Linear Regression Model)

Suppose we observe $n$ independent observations with $p$ predictors $(x_{i 1}, x_{i 2}, \dots, x_{i p}, y_i),\quad i = 1, \dots, n$
For the $i$ th observation, the multiple linear regression model is $y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip} + \varepsilon_i, \quad i = 1, \dots, n.$
In matrix form, $\boldsymbol y = \boldsymbol X \boldsymbol\beta + \boldsymbol\varepsilon$
where $\boldsymbol y = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}, \quad \boldsymbol X = \begin{pmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{pmatrix}, \quad \boldsymbol\beta = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{pmatrix}, \quad \boldsymbol\varepsilon = \begin{pmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{pmatrix}$

2. Basic Assumptions

Definition (Basic Assumptions)

Consider the multiple linear regression model $\boldsymbol y = \boldsymbol X \boldsymbol\beta + \boldsymbol\varepsilon$ . The basic assumptions are:

Zero mean: $E(\boldsymbol\varepsilon) = \boldsymbol 0$ , or equivalently $E(\boldsymbol y) = \boldsymbol X \boldsymbol\beta$ .
Constant variance and uncorrelated errors: $\operatorname{Var}(\boldsymbol\varepsilon) = \sigma^2 I_n$ , i.e. $\operatorname{Var}(\varepsilon_i) = \sigma^2$ and $\operatorname{Cov}(\varepsilon_i, \varepsilon_j) = 0$ for $i \ne j$ .
Normality (for exact finite-sample inference): $\boldsymbol\varepsilon \sim N(\boldsymbol 0, \sigma^2 I_n)$ .

3. Estimation

3.1 Ordinary Least Squares

The ordinary least squares (OLS) estimator minimizes the sum of squared residuals

S(\boldsymbol\beta) = (\boldsymbol y - \boldsymbol X\boldsymbol\beta)^\top(\boldsymbol y - \boldsymbol X\boldsymbol\beta).

Expanding,

S(\boldsymbol\beta) = \boldsymbol y^\top\boldsymbol y - 2\boldsymbol\beta^\top\boldsymbol X^\top\boldsymbol y + \boldsymbol\beta^\top\boldsymbol X^\top\boldsymbol X\boldsymbol\beta.

Taking derivative with respect to $\boldsymbol\beta$ ,

\frac{\partial S(\boldsymbol\beta)}{\partial \boldsymbol\beta} = -2\boldsymbol X^\top\boldsymbol y + 2\boldsymbol X^\top\boldsymbol X\boldsymbol\beta.

Setting the derivative equal to zero gives the normal equations

\boldsymbol X^\top \boldsymbol X \hat{\boldsymbol\beta} = \boldsymbol X^\top \boldsymbol y.

Since the Hessian $2\boldsymbol X^\top\boldsymbol X$ is positive semidefinite, the solution minimizes $S$ . If $\operatorname{rank}(\boldsymbol X) = p + 1$ , then $\boldsymbol X^\top\boldsymbol X$ is nonsingular and

\hat{\boldsymbol\beta} = (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol y.

3.2 Fitted Values and Residuals

The fitted values are

\hat{\boldsymbol y} = \boldsymbol X\hat{\boldsymbol\beta} = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol y = \boldsymbol H\boldsymbol y,

where

\boldsymbol H = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top

is called the hat matrix. The residual vector is

\hat{\boldsymbol\varepsilon} = \boldsymbol y - \hat{\boldsymbol y} = (I_n - \boldsymbol H)\boldsymbol y.

Moreover, from the normal equations,

\boldsymbol X^\top\hat{\boldsymbol\varepsilon} = \boldsymbol X^\top\boldsymbol y - \boldsymbol X^\top\boldsymbol X\hat{\boldsymbol\beta} = \boldsymbol 0,

i.e., each column of $\boldsymbol X$ is orthogonal to $\hat{\boldsymbol\varepsilon}$ .

3.3 Properties of the Hat Matrix

Theorem (Properties of the Hat Matrix)

The hat matrix $\boldsymbol H = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top$ has the following important properties:

Symmetric: $\boldsymbol H^\top = \boldsymbol H$ .
Idempotent: $\boldsymbol H^2 = \boldsymbol H$ .
Orthogonal projection: $\hat{\boldsymbol y} = \boldsymbol H\boldsymbol y \in \mathcal C(\boldsymbol X)$ , $\hat{\boldsymbol\varepsilon} = (I_n - \boldsymbol H)\boldsymbol y \perp \mathcal C(\boldsymbol X)$ .
Rank and trace: $\operatorname{rank}(\boldsymbol H) = \operatorname{tr}(\boldsymbol H) = p + 1$ .

Thus $\boldsymbol H$ is the orthogonal projection matrix onto $\mathcal C(\boldsymbol X)$ , the column space of $\boldsymbol X$ .

Proof.

Since $(\boldsymbol X^\top\boldsymbol X)^{-1}$ is symmetric,
$\boldsymbol H^\top = \boldsymbol X\left[(\boldsymbol X^\top\boldsymbol X)^{-1}\right]^\top\boldsymbol X^\top = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top = \boldsymbol H.$
$\boldsymbol H^2 = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}(\boldsymbol X^\top\boldsymbol X)(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top = \boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top = \boldsymbol H.$
$\hat{\boldsymbol y} = \boldsymbol X\hat{\boldsymbol\beta}$ is a linear combination of the columns of $\boldsymbol X$ , so $\hat{\boldsymbol y} \in \mathcal C(\boldsymbol X)$ . Moreover,
$\boldsymbol X^\top\boldsymbol H = \boldsymbol X^\top\boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top = \boldsymbol X^\top,$
so $\boldsymbol X^\top(I_n - \boldsymbol H) = \boldsymbol 0$ , i.e., $\hat{\boldsymbol\varepsilon} \perp \mathcal C(\boldsymbol X)$ .
A symmetric idempotent matrix has all eigenvalues equal to $0$ or $1$ , so $\operatorname{rank}(\boldsymbol H) = \operatorname{tr}(\boldsymbol H)$ . By cyclicity of the trace,
$\operatorname{tr}(\boldsymbol H) = \operatorname{tr}\left[(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol X\right] = \operatorname{tr}(I_{p+1}) = p + 1.$

3.4 Estimation of Error Variance

Recall that $\hat{\boldsymbol\varepsilon} = (I_n - \boldsymbol H)\boldsymbol y$ . Since $(I_n - \boldsymbol H)\boldsymbol X\boldsymbol\beta = \boldsymbol 0$ ,

\hat{\boldsymbol\varepsilon} = (I_n - \boldsymbol H)\boldsymbol\varepsilon.

The residual sum of squares is $SSE = \hat{\boldsymbol\varepsilon}^\top\hat{\boldsymbol\varepsilon}$ . Since $I_n - \boldsymbol H$ is symmetric and idempotent,

E(\hat{\boldsymbol\varepsilon}^\top\hat{\boldsymbol\varepsilon}) = \sigma^2\operatorname{tr}(I_n - \boldsymbol H) = \sigma^2\{n - (p + 1)\}.

Proof. By symmetry and idempotency of $I_n - \boldsymbol H$ ,

\hat{\boldsymbol\varepsilon}^\top\hat{\boldsymbol\varepsilon} = \boldsymbol\varepsilon^\top(I_n - \boldsymbol H)^\top(I_n - \boldsymbol H)\boldsymbol\varepsilon = \boldsymbol\varepsilon^\top(I_n - \boldsymbol H)\boldsymbol\varepsilon.

For any fixed symmetric matrix $A$ , since $E(\boldsymbol\varepsilon) = \boldsymbol 0$ and $\operatorname{Var}(\boldsymbol\varepsilon) = \sigma^2 I_n$ ,

E(\boldsymbol\varepsilon^\top A\boldsymbol\varepsilon) = E\left[\operatorname{tr}\left(A\boldsymbol\varepsilon\boldsymbol\varepsilon^\top\right)\right] = \sigma^2\operatorname{tr}(A).

Taking $A = I_n - \boldsymbol H$ and using $\operatorname{tr}(\boldsymbol H) = p + 1$ gives the result.

Definition (Unbiased Estimator of $\sigma^2$ )

An unbiased estimator of $\sigma^2$ is $\hat\sigma^2 = MSE = \frac{SSE}{n - p - 1}.$

3.5 Maximum Likelihood Estimation

Under the normal linear model $\boldsymbol y \mid \boldsymbol X \sim N(\boldsymbol X\boldsymbol\beta, \sigma^2 I_n)$ , the likelihood is

L(\boldsymbol\beta, \sigma^2) = (2\pi\sigma^2)^{-n/2}\exp\left\{-\frac{1}{2\sigma^2}(\boldsymbol y - \boldsymbol X\boldsymbol\beta)^\top(\boldsymbol y - \boldsymbol X\boldsymbol\beta)\right\}.

Maximizing the likelihood gives

\hat{\boldsymbol\beta}_{MLE} = (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol y = \hat{\boldsymbol\beta}_{OLS}

and

\hat\sigma^2_{MLE} = \frac{SSE}{n}.

Proof. For fixed $\sigma^2$ , maximizing $L$ over $\boldsymbol\beta$ is equivalent to minimizing $(\boldsymbol y - \boldsymbol X\boldsymbol\beta)^\top(\boldsymbol y - \boldsymbol X\boldsymbol\beta)$ , which yields the OLS estimator. The log-likelihood is, up to constants,

\ell(\boldsymbol\beta, \sigma^2) = -\frac{n}{2}\log\sigma^2 - \frac{S(\boldsymbol\beta)}{2\sigma^2}.

Substituting $\hat{\boldsymbol\beta}$ and solving $\partial\ell/\partial\sigma^2 = -\dfrac{n}{2\sigma^2} + \dfrac{SSE}{2\sigma^4} = 0$ gives $\hat\sigma^2_{MLE} = SSE/n$ . Note that $\hat\sigma^2_{MLE}$ is biased downward, since $E(SSE) = \sigma^2(n - p - 1)$ .

4. Properties of the OLS Estimator

4.1 Unbiasedness and Variance

Theorem (Unbiasedness and Variance)

Under Assumptions 1–2, the OLS estimator satisfies $E(\hat{\boldsymbol\beta}) = \boldsymbol\beta, \qquad \operatorname{Var}(\hat{\boldsymbol\beta}) = \sigma^2(\boldsymbol X^\top\boldsymbol X)^{-1}.$

Proof. Recall that

\hat{\boldsymbol\beta} = (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol y = \boldsymbol\beta + (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol\varepsilon.

Therefore $E(\hat{\boldsymbol\beta}) = \boldsymbol\beta + (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top E(\boldsymbol\varepsilon) = \boldsymbol\beta$ . Moreover,

\operatorname{Var}(\hat{\boldsymbol\beta}) = (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top(\sigma^2 I_n)\boldsymbol X(\boldsymbol X^\top\boldsymbol X)^{-1} = \sigma^2(\boldsymbol X^\top\boldsymbol X)^{-1}.

4.2 Distributional Properties under Normality

Theorem (Distributional Properties under Normality)

If $\boldsymbol y \mid \boldsymbol X \sim N(\boldsymbol X\boldsymbol\beta, \sigma^2 I_n)$ , then:

$\hat{\boldsymbol\beta} \sim N\left(\boldsymbol\beta, \sigma^2(\boldsymbol X^\top\boldsymbol X)^{-1}\right)$ .
$\dfrac{SSE}{\sigma^2} \sim \chi^2_{n-p-1}$ .
$\hat{\boldsymbol\beta}$ and $\hat{\boldsymbol\varepsilon}$ are independent, hence $\hat{\boldsymbol\beta}$ is independent of $SSE$ (and $\hat\sigma^2$ ).

Proof.

$\hat{\boldsymbol\beta} = \boldsymbol\beta + (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top\boldsymbol\varepsilon$ is an affine transformation of the multivariate normal vector $\boldsymbol\varepsilon$ , hence normal with mean $\boldsymbol\beta$ and variance $\sigma^2(\boldsymbol X^\top\boldsymbol X)^{-1}$ .
Recall that $\hat{\boldsymbol\varepsilon} = (I_n - \boldsymbol H)\boldsymbol\varepsilon$ , so
$\frac{SSE}{\sigma^2} = \left(\frac{\boldsymbol\varepsilon}{\sigma}\right)^\top(I_n - \boldsymbol H)\left(\frac{\boldsymbol\varepsilon}{\sigma}\right),$
a quadratic form in $\boldsymbol\varepsilon/\sigma \sim N(\boldsymbol 0, I_n)$ . A standard result states that $\boldsymbol Z^\top A\boldsymbol Z \sim \chi^2_r$ if $\boldsymbol Z \sim N(\boldsymbol 0, I_n)$ and $A$ is symmetric idempotent with $\operatorname{rank}(A) = r$ (diagonalize $A$ by an orthogonal matrix). Here $A = I_n - \boldsymbol H$ has rank $n - p - 1$ .
$\hat{\boldsymbol\beta}$ and $\hat{\boldsymbol\varepsilon}$ are jointly normal as linear transformations of $\boldsymbol y$ . Since $\boldsymbol X^\top\boldsymbol H = \boldsymbol X^\top$ ,
$\operatorname{Cov}(\hat{\boldsymbol\beta}, \hat{\boldsymbol\varepsilon}) = (\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol X^\top \cdot \sigma^2 I_n \cdot (I_n - \boldsymbol H) = \sigma^2(\boldsymbol X^\top\boldsymbol X)^{-1}(\boldsymbol X^\top - \boldsymbol X^\top\boldsymbol H) = \boldsymbol 0.$
Under joint normality, zero covariance implies independence.

Note (Property 3 is what makes the $t$ -statistics and confidence intervals below exact in finite samples.)

5. Test

5.1 Overall F Test

Consider testing whether there is a regression relation between the response variable and the set of predictors:

H_0: \beta_1 = \beta_2 = \cdots = \beta_p = 0, \quad H_1: \text{at least one } \beta_j \ne 0.

The total variation can be decomposed as

SST = SSR + SSE,

where

SST = \sum_{i=1}^n (y_i - \bar y)^2, \qquad SSE = \sum_{i=1}^n (y_i - \hat y_i)^2, \qquad SSR = \sum_{i=1}^n (\hat y_i - \bar y)^2.

Proof of the decomposition. Since $y_i - \bar y = \hat\varepsilon_i + (\hat y_i - \bar y)$ ,

SST = SSE + SSR + 2\sum_{i=1}^n \hat\varepsilon_i(\hat y_i - \bar y).

The cross term equals $\hat{\boldsymbol\varepsilon}^\top\hat{\boldsymbol y} - \bar y\,\hat{\boldsymbol\varepsilon}^\top\boldsymbol 1$ . Both $\hat{\boldsymbol y}$ and $\boldsymbol 1$ (the first column of $\boldsymbol X$ ) lie in $\mathcal C(\boldsymbol X)$ , while $\hat{\boldsymbol\varepsilon} \perp \mathcal C(\boldsymbol X)$ , so the cross term vanishes.

Define

MSR = \frac{SSR}{p}, \qquad MSE = \frac{SSE}{n - p - 1}.

Under $H_0$ ,

F = \frac{MSR}{MSE} \sim F_{p, n-p-1};

reject $H_0$ for large values of $F$ .

5.2 Test for an Individual Regression Coefficient

To test

H_0: \beta_j = \beta_{j0}, \quad H_1: \beta_j \ne \beta_{j0},

let $C = (\boldsymbol X^\top\boldsymbol X)^{-1}$ . Since $\operatorname{Var}(\hat\beta_j) = \sigma^2 C_{jj}$ , the standard error is estimated by $\widehat{SE}(\hat\beta_j) = \hat\sigma\sqrt{C_{jj}}$ , and the test statistic is

t = \frac{\hat\beta_j - \beta_{j0}}{\hat\sigma\sqrt{C_{jj}}}.

Under $H_0$ , $t \sim t_{n-p-1}$ , since $\hat\beta_j \sim N(\beta_j, \sigma^2 C_{jj})$ is independent of $SSE/\sigma^2 \sim \chi^2_{n-p-1}$ . For the common hypothesis $H_0: \beta_j = 0$ ,

t = \frac{\hat\beta_j}{\widehat{SE}(\hat\beta_j)}.

Reject $H_0$ when $|t|$ is sufficiently large.

5.3 Interpretation of the Individual t-Test

For $H_0: \beta_j = 0$ , the question is whether $X_j$ contributes to the regression after controlling for the other predictors. In the multiple regression model

E(Y \mid X_1, \dots, X_p) = \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p,

$\beta_j$ measures the change in the conditional mean of $Y$ associated with a one-unit increase in $X_j$ , holding all other predictors fixed. Thus

H_0: \beta_j = 0

tests whether $X_j$ provides additional linear information given the other predictors in the model.

5.4 Partial F-Test

Suppose we want to test whether a group of $q$ predictors contributes to the model:

H_0: \beta_{j_1} = \cdots = \beta_{j_q} = 0.

Compare the reduced model ( $H_0$ imposed) with the full model (all predictors included). Let $SSE_R$ be the SSE from the reduced model and $SSE_F$ the SSE from the full model. The extra sum of squares due to the $q$ predictors is

SSE_R - SSE_F,

and the partial F statistic is

F = \frac{(SSE_R - SSE_F)/q}{SSE_F/(n - p - 1)}.

Under $H_0$ , $F \sim F_{q, n-p-1}$ ; reject $H_0$ for large values of $F$ .

6. Confidence Intervals

6.1 Confidence Intervals for Regression Coefficients

Let $C = (\boldsymbol X^\top\boldsymbol X)^{-1}$ and $\hat\sigma^2 = SSE/(n - p - 1)$ . Since

\frac{\hat\beta_j - \beta_j}{\hat\sigma\sqrt{C_{jj}}} \sim t_{n-p-1},

a $100(1-\alpha)\%$ confidence interval for $\beta_j$ is

\hat\beta_j \pm t_{1-\alpha/2,\,n-p-1}\,\hat\sigma\sqrt{C_{jj}}.

More generally, for any linear combination $\theta = \boldsymbol a^\top\boldsymbol\beta$ , the estimator $\hat\theta = \boldsymbol a^\top\hat{\boldsymbol\beta}$ is normal (as a linear combination of the jointly normal $\hat{\boldsymbol\beta}$ ) and independent of $SSE$ , so a $100(1-\alpha)\%$ confidence interval for $\theta$ is

\boldsymbol a^\top\hat{\boldsymbol\beta} \pm t_{1-\alpha/2,\,n-p-1}\,\hat\sigma\sqrt{\boldsymbol a^\top(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol a}.

6.2 Confidence Interval for the Mean Response

For a given predictor vector

\boldsymbol x_0 = (1, x_{01}, \dots, x_{0p})^\top,

the conditional mean response is

\mu_0 = E(Y_0 \mid \boldsymbol x_0) = \boldsymbol x_0^\top\boldsymbol\beta,

with estimator

\hat\mu_0 = \boldsymbol x_0^\top\hat{\boldsymbol\beta}.

Since

\operatorname{Var}(\hat\mu_0 \mid \boldsymbol X) = \sigma^2\,\boldsymbol x_0^\top(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol x_0,

a $100(1-\alpha)\%$ confidence interval is

\hat\mu_0 \pm t_{1-\alpha/2,\,n-p-1}\,\hat\sigma\sqrt{\boldsymbol x_0^\top(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol x_0}.

The target here is the average response of the population at $\boldsymbol x_0$ .

6.3 Prediction Interval for a New Observation

For a new observation at the same predictor value $\boldsymbol x_0$ ,

Y_0 = \boldsymbol x_0^\top\boldsymbol\beta + \varepsilon_0.

The point prediction is

\hat Y_0 = \boldsymbol x_0^\top\hat{\boldsymbol\beta} = \hat\mu_0.

Thus the two point predictions are the same, but their uncertainties are different. The prediction error is

Y_0 - \hat Y_0 = \boldsymbol x_0^\top(\boldsymbol\beta - \hat{\boldsymbol\beta}) + \varepsilon_0,

where $\varepsilon_0$ is independent of $\hat{\boldsymbol\beta}$ , so

\operatorname{Var}(Y_0 - \hat Y_0 \mid \boldsymbol X, \boldsymbol x_0) = \sigma^2\left[1 + \boldsymbol x_0^\top(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol x_0\right].

Hence a $100(1-\alpha)\%$ prediction interval is

\hat Y_0 \pm t_{1-\alpha/2,\,n-p-1}\,\hat\sigma\sqrt{1 + \boldsymbol x_0^\top(\boldsymbol X^\top\boldsymbol X)^{-1}\boldsymbol x_0}.

The extra $+1$ under the square root reflects the uncertainty of the new noise term $\varepsilon_0$ , so the prediction interval is always wider than the confidence interval for the mean response.

7. Residuals

Recall that

\hat{\boldsymbol\varepsilon} = \boldsymbol y - \hat{\boldsymbol y} = (I_n - \boldsymbol H)\boldsymbol y.

Then

E(\hat{\boldsymbol\varepsilon}) = \boldsymbol 0 \quad \text{and} \quad \operatorname{Var}(\hat{\boldsymbol\varepsilon}) = \sigma^2(I_n - \boldsymbol H),

so for the $i$ th residual, with $h_{ii}$ the $i$ th diagonal element of the hat matrix,

\operatorname{Var}(\hat\varepsilon_i) = \sigma^2(1 - h_{ii}).

Proof. $E(\hat{\boldsymbol\varepsilon}) = (I_n - \boldsymbol H)\boldsymbol X\boldsymbol\beta = \boldsymbol 0$ , and by symmetry and idempotency of $I_n - \boldsymbol H$ ,

\operatorname{Var}(\hat{\boldsymbol\varepsilon}) = (I_n - \boldsymbol H)\,\sigma^2 I_n\,(I_n - \boldsymbol H) = \sigma^2(I_n - \boldsymbol H).

7.1 Studentized Residuals

Because $\operatorname{Var}(\hat\varepsilon_i) = \sigma^2(1 - h_{ii})$ varies with $i$ , raw residuals are not directly comparable across observations. The studentized residual is

r_i = \frac{\hat\varepsilon_i}{\hat\sigma\sqrt{1 - h_{ii}}},

which adjusts the residual by its estimated standard deviation.