Biostatistics Study NotesFrom concepts and derivations to interpretation

CHAPTER 07 · TOPIC 01

Simple Linear Regression

Learn least squares, residual diagnostics, sums-of-squares decomposition, coefficient tests, and prediction with one quantitative predictor.

On this page
  1. Questions Simple Linear Regression Can Answer
  2. Start with the Scatterplot and Residuals
  3. Population Model and Fitted Sample Line
  4. Interpreting the Intercept and Slope
  5. Least Squares
  6. Model Assumptions
  7. Decomposing Variation in Y into Model and Error
  8. Regression ANOVA Table and Overall F Test
  9. The reason why the degrees of freedom are n−1, 1, and n−2
  10. Residual Standard Deviation and Coefficient Tests
  11. F=t² with slope test
  12. Distinguishing Explanation from Prediction

Simple regression analysis models a linear relationship between one quantitative explanatory variable, X, and a quantitative outcome variable, Y. The slope represents the direction and magnitude of the association and is used to explain the average response and predict new observations.

Questions Simple Linear Regression Can Answer#

  • How much does the average of Y change when X increases by 1 unit?
  • Is there evidence that the linear relationship is different from 0?
  • How much variation in Y does the model explain?
  • Can a given X predict the average response or individual values?

Start with the Scatterplot and Residuals#

Look first at curvature, clusters, outliers, high-leverage points, and changes in spread. The difference between the observed value at each point and the regression line is the residual.

ei=Yi−Y^ie_i=Y_i-\hat Y_i

Population Model and Fitted Sample Line#

population modelYi=β0+β1Xi+εiY_i=\beta_0+\beta_1X_i+\varepsilon_i
Population mean at a fixed XE(Yi∣Xi)=β0+β1XiE(Y_i\mid X_i)=\beta_0+\beta_1X_i
fitted sample lineY^i=b0+b1Xi\hat Y_i=b_0+b_1X_i
residualei=Yi−Y^ie_i=Y_i-\hat Y_i

Interpreting the Intercept and Slope#

β₁ is the amount of change in Y average when X increases by 1 unit. β₀ is the average at X=0, but if 0 is outside the observation range or has no practical meaning, do not force it to be interpreted. Centering allows you to move the intercept within the observation range.

Least Squares#

(b0,b1)=arg⁡min⁡a,c∑i=1n[Yi−(a+cXi)]2(b_0,b_1)=\arg\min_{a,c}\sum_{i=1}^{n}[Y_i-(a+cX_i)]^2
slopeb1=∑(Xi−Xˉ)(Yi−Yˉ)∑(Xi−Xˉ)2b_1=\frac{\sum(X_i-\bar X)(Y_i-\bar Y)}{\sum(X_i-\bar X)^2}
interceptb0=Yˉ−b1Xˉb_0=\bar Y-b_1\bar X

Model Assumptions#

  • The mean of Y is linearly related to X
  • Observations and errors are independent
  • The error variance is constant across levels of X
  • For small-sample inference on coefficients, the errors are approximately normal
  • Strong outliers and influence points do not dominate the results
mean errorE(εi∣Xi)=0E(\varepsilon_i\mid X_i)=0
equal varianceVar⁡(εi∣Xi)=σ2\operatorname{Var}(\varepsilon_i\mid X_i)=\sigma^2
normal errorεi∣Xi∼N(0,σ2)\varepsilon_i\mid X_i\sim N(0,\sigma^2)

Decomposing Variation in Y into Model and Error#

Yi−Yˉ⏟Total deviation=Y^i−Yˉ⏟Explained by regression+Yi−Y^i⏟Residual\underbrace{Y_i-\bar Y}_{\text{Total deviation}}=\underbrace{\hat Y_i-\bar Y}_{\text{Explained by regression}}+\underbrace{Y_i-\hat Y_i}_{\text{Residual}}
total sum of squaresSST=∑(Yi−Yˉ)2SS_T=\sum(Y_i-\bar Y)^2
regression sum of squaresSSR=∑(Y^i−Yˉ)2=b12∑(Xi−Xˉ)2SS_R=\sum(\hat Y_i-\bar Y)^2=b_1^2\sum(X_i-\bar X)^2
error sum of squaresSSE=∑(Yi−Y^i)2SS_E=\sum(Y_i-\hat Y_i)^2
DecompositionSST=SSR+SSESS_T=SS_R+SS_E

Regression ANOVA Table and Overall F Test#

source of variationsum of squaresdegrees of freedommean square
regressionSSR1SSR/1
errorSSEn−2SSE/(n−2)
totalSSTn−1—
null hypothesisH0:β1=0H_0:\beta_1=0
F-statisticF=SSR/1SSE/(n−2)F=\frac{SS_R/1}{SS_E/(n-2)}
H₀bottomF∼F1,n−2F\sim F_{1,n-2}

The reason why the degrees of freedom are n−1, 1, and n−2#

The total deviation uses one degrees of freedom to estimate the mean. The regression part is determined by one slope and has 1 degrees of freedom, and the residual has n-2 degrees of freedom because it estimates two parameters: the intercept and the slope.

Residual sum∑ei=0\sum e_i=0
Orthogonal to X∑(Xi−Xˉ)ei=0\sum(X_i-\bar X)e_i=0
error degrees of freedomdfE=n−2df_E=n-2

Residual Standard Deviation and Coefficient Tests#

residual standard deviationSY∣X=SSEn−2S_{Y\mid X}=\sqrt{\frac{SS_E}{n-2}}
sum of squares of xSxx=∑(Xi−Xˉ)2S_{xx}=\sum(X_i-\bar X)^2
standard error of interceptSE(b0)=SY∣X1n+Xˉ2SxxSE(b_0)=S_{Y\mid X}\sqrt{\frac1n+\frac{\bar X^2}{S_{xx}}}
standard error of slopeSE(b1)=SY∣XSxxSE(b_1)=\frac{S_{Y\mid X}}{\sqrt{S_{xx}}}
intercept testt=b0−c0SE(b0)t=\frac{b_0-c_0}{SE(b_0)}
slope testt=b1−c1SE(b1),df=n−2t=\frac{b_1-c_1}{SE(b_1)},\qquad df=n-2

F=t² with slope test#

t squaredt2=b12SxxSY∣X2t^2=\frac{b_1^2S_{xx}}{S_{Y\mid X}^2}
numeratorb12Sxx=SSRb_1^2S_{xx}=SS_R
denominatorSY∣X2=MSES_{Y\mid X}^2=MS_E
conclusiont2=SSR/1SSE/(n−2)=Ft^2=\frac{SS_R/1}{SS_E/(n-2)}=F

Distinguishing Explanation from Prediction#

The confidence interval for the mean response given X represents the uncertainty in the population mean line. The prediction interval for a new individual will always be wider because individual error is also added. Extrapolation outside the observation range should be handled with caution as there is no guarantee of a linear relationship.

  1. Check the scatter plot and study design
  2. Estimate coefficients and confidence intervals
  3. Diagnose residuals, influence points, linearity, and homoscedasticity
  4. Report F, slope t, p-value, R², residual standard deviation
  5. Clarify the prediction target and scope of application without interpreting association as causation