Multiple Regression | Regression Analysis | Biostatistics Study Notes

Multiple regression analysis puts multiple explanatory variables into the same linear model and estimates the conditional association of each variable, holding other variables constant. Used for prediction, confounding adjustment, and simultaneous comparison of multiple factors.

population model

Y_i=\beta_0+\beta_1X_{1i}+\cdots+\beta_pX_{pi}+\varepsilon_i

sample prediction formula

\hat Y_i=b_0+b_1X_{1i}+\cdots+b_pX_{pi}

Interpreting Multiple Regression Coefficients#

E(\text{Weight})=\beta_0+\beta_1(\text{Age})+\beta_2(\text{Female})+\beta_3(\text{Height})

β₁ is the average weight difference associated with a 1 unit increase in age when comparing subjects of the same gender and height. β₂ is the average difference between women and the reference gender, holding age and height constant. The important difference from simple regression is the conditional interpretation of ``holding other things constant''.

Decomposing Variation in Y#

Deviation decomposition

Y_i-\bar Y=(\hat Y_i-\bar Y)+(Y_i-\hat Y_i)

total sum of squares

SS_T=\sum(Y_i-\bar Y)^2

regression sum of squares

SS_R=\sum(\hat Y_i-\bar Y)^2

error sum of squares

SS_E=\sum(Y_i-\hat Y_i)^2

Decomposition

SS_T=SS_R+SS_E

Overall F test#

H_0:\beta_1=\beta_2=\cdots=\beta_p=0\qquad H_1:\text{At least one }\beta_j\ne0

source of variation	sum of squares	degrees of freedom	mean square
regression	SSR	p	SSR/p
error	SSE	n−p−1	SSE/(n−p−1)
total	SST	n−1	—

regression mean square

MS_R=\frac{SS_R}{p}

error mean square

MS_E=\frac{SS_E}{n-p-1}

F-statistic

F=\frac{SS_R/p}{SS_E/(n-p-1)}

H₀bottom

F\sim F_{p,n-p-1}

Tests for Individual Coefficients#

individual hypothesis

H_0:\beta_j=0\qquad H_1:\beta_j\ne0

t-statistic

t=\frac{b_j}{SE(b_j)},\qquad df=n-p-1

Even if the overall F is significant, it does not necessarily mean that all coefficients are significant, and even if the individual t is not significant, it may be useful as a group of variables. For joint hypotheses with multiple coefficients, partial F tests and Wald tests are used.

Relationship between Categorical Predictors and ANOVA#

dummy variables for J groups

p=J-1

error degrees of freedom

n-p-1=n-J

A factor with J levels is represented by J−1 dummy variables, and one level is used as the reference group. The overall F test for a factor-only regression model is the same as a one-way ANOVA.

Coding of dummy variables#

group	X₁	X₂	average
reference group	0	0	β₀
group 2	1	0	β₀+β₁
group 3	0	1	β₀+β₂

E(Y)=\beta_0+\beta_1X_1+\beta_2X_2

β1 and β2 are the differences from the reference group, respectively. Use appropriate contrasts for comparisons between different groups and overall effects, and consider multiplicity.

Why Include Confounders in the Model?#

unadjusted

E(\text{Vision})=\beta_0+\beta_1(\text{Wealth})

age adjustment

E(\text{Vision})=\beta_0+\beta_1(\text{Wealth})+\beta_2(\text{Age})

Unadjusted coefficients can be confounded if age is related to both wealth and vision. β₁ with age is the conditional association at the same age. However, we also check linearity, interactions, measurement error, and overlap.

Crude, Partially Adjusted, and Fully Adjusted Models#

Model	Purpose
Crude	Unadjusted association between exposure and outcome
Partially adjusted	Add key prespecified confounders and examine changes in the coefficient
Fully adjusted	Include variables and functional forms specified by the causal framework and analysis plan

Model Assumptions and Diagnostics#

Mean structure is appropriate for each continuous variable (nonlinear terms if necessary)
Observation and error are independent
Residual variance is approximately constant
Residuals are generally normal in small sample inference
Avoid severe multicollinearity; examine the VIF and condition number
Assess uncertainty in influence points, missing data, extrapolation, and model selection

Analysis and Reporting Workflow#

Predefine research questions, estimation targets, and variable selection
Check distribution, missingness, correlation, and encoding
Specify linearity and interactions in the form required for research
Estimate overall F, coefficients, 95% confidence intervals, p-values, R² and adjusted R²
Diagnose residuals, multicollinearity, influence points, and predictive performance
Specify unadjusted/adjusted results, units, reference groups, and limits