Post Hoc Comparisons | Analysis of Variance

Even if the overall ANOVA is significant, it only tells us that at least one population mean is different; it does not tell us which groups are different. To determine location, use pre-planned contrasts or post hoc comparisons.

Why Not Repeat t Tests?#

For comparisons, decide how to estimate the standard error of the mean difference and how to control the significance level after multiple tests. The total number of pairs in group k is as follows.

m=\binom{k}{2}=\frac{k(k-1)}{2}

If there are 3 groups, there will be 3 comparisons, and if there are 5 groups, there will be 10 comparisons. Even if we perform each test with α=0.05, the probability of producing at least one false positive across the comparison family remains greater than 0.05.

Per-Comparison Error and Family-Wise Error#

No type 1 errors in all m times

P(\text{No type 1 error})=(1-\alpha_1)^m

At least one Type I error

\alpha_F=1-(1-\alpha_1)^m

When α₁ is small

\alpha_F\approx m\alpha_1

α₁ is the type I error rate for each comparison, and αF is the family-wise error rate (FWER) of making at least one type I error in one comparison group. The intuition in the above equation is based on independent tests, but the Bonferroni guarantee does not require independence between tests.

Common Standard Error for a Two-Group Mean Difference#

For one-way ANOVA with equal variances, you can use the MSE in the ANOVA table as the common error variance for each pair.

standard error of mean difference

SE(\bar X_i-\bar X_j)=\sqrt{MS_E\left(\frac1{n_i}+\frac1{n_j}\right)}

Paired comparison t-statistic

t_{ij}=\frac{\bar X_i-\bar X_j}{\sqrt{MS_E\left(\frac1{n_i}+\frac1{n_j}\right)}}

degree of freedom

df=n-k

Fisher's LSD method#

Fisher's least significant difference method is a pooled t-type pairwise comparison using the MSE of ANOVA. Normally, if the overall ANOVA is significant, each comparison is also judged by the original α, without making any additional adjustments depending on the number of comparisons.

p_{ij}<\alpha\quad\Longrightarrow\quad\text{reject }H_{0,ij}

Bonferroni method#

Allocate the target family significance level αF to m comparisons. The method of determining each comparison using αF/m and the method of multiplying the original p-value by m and comparing it with αF are equivalent.

Significance level for each comparison

\alpha_{\mathrm{per\ comparison}}=\frac{\alpha_F}{m}

Adjusted p-value

p_{\mathrm{adj}}=\min(mp,1)

If αF=0.05 for all 3 comparisons of 3 groups, each threshold is 0.05/3≈0.0167. It is simple and controls FWER regardless of the dependency structure, but the more comparisons there are, the more conservative it becomes, which may increase Type II error.

Holm method#

Holm's step-down method serializes Bonferroni and typically yields higher power while controlling FWER.

Sort m p-values into p(1)≤p(2)≤⋯≤p(m)
Compare the minimum p-value with αF/m. If it is not established, it will be stopped and will not be rejected thereafter.
If it holds true, compare the next with αF/(m−1) and loosen the gate according to the number of remaining comparisons.
Stops at the first failure, and does not reject subsequent terms.

p_{(i)}\leq\frac{\alpha_F}{m-i+1}

p_{\mathrm{Holm},(i)}=\min\!\left(1,\max_{1\leq j\leq i}[(m-j+1)p_{(j)}]\right)

The adjusted p-value should be monotonic so that a larger original p-value does not become a smaller adjusted value.

Holm–Šidák method#

The step-down procedure of arranging the p-values in descending order is the same, but we use the Šidák gate for the remaining r hypotheses.

number of comparisons remaining

r=m-i+1

Stage i gate

\alpha_i'=1-(1-\alpha_F)^{1/r}

Choosing among the Four Methods#

method	Handling multiplicity	Features
Fisher's LSD	Usually no adjustment	High detection power, but FWER tends to increase
Bonferroni	Divide αF by m	Simple, robust, but conservative
Holm	Gradual mitigation of Bonferroni barriers	Controls FWER and has higher detection power than normal Bonferroni
Holm–Šidák	Sequential Šidák gates	Slightly lenient, but be careful of dependencies