Degrees of Freedom | Probability Distributions

Degrees of freedom (df) represents how many pieces of information remain in the data that can change freely and independently under given conditions and constraints. It is not the size of the data or the number of data that can be arbitrarily deleted, but rather the amount of independent information that actually remains when calculating statistics.

Understand with an example where the average value is fixed#

Suppose we know that the average of 4 pieces of data is 10. The sum of the 4 values must be 40. The first three values can be freely selected, but the fourth value cannot be changed.

Constraints by average value

\frac{x_1+x_2+x_3+x_4}{4}=10\quad\Longrightarrow\quad x_1+x_2+x_3+x_4=40

The first three values determine the fourth

x_4=40-(x_1+x_2+x_3)

Amount of information that can be changed freely

df=4-1=3

For example, if the first three values are 8, 11, and 13, the fourth value will be 8 to keep the average at 10. There are 4 pieces of data, but only 3 pieces can be freely determined.

symbol	meaning
n	Number of data or sample size
df	Degree of freedom. numbers that give independent information
xᵢ	i-th observation
x̄	sample mean
n−1	degrees of freedom remaining after estimating one mean

Understanding degrees of freedom from linear algebra#

Degrees of freedom are closely related to "dimensions" and "independent directions" in linear algebra. If there are no constraints on the n values, they can change in n independent directions and have n degrees of freedom. Adding one independent constraint makes one direction unfree, usually n-1.

In the previous example, the 4 values can originally change in 4 directions, but if the total is fixed to 40, only one direction is restricted. Although there are still four values, there are only three directions of change that are independent of each other.

Therefore, what is important is not only the number of constraint expressions, but also the number of mutually independent constraints. Constraints that express the same content or that can be derived from other constraints do not reduce the degree of freedom by one more.

Why is the sample variance divided by n-1?#

When calculating the sample variance, first estimate the sample mean from the same data. The sum of n deviations from the mean must be 0, and knowing the first n-1 deviations determines the last deviation. Since the degrees of freedom for the deviation are n-1, we also use n-1 for the denominator of the sample variance.

Constraints on deviation

\sum_{i=1}^{n}(x_i-\bar{x})=0

sample variance

s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2

Degrees of Freedom Are Not Always n−1#

The degrees of freedom are determined by the amount of independent information, constraints, and the number of estimated parameters. Because each statistical method has a different formula, you don't always look at the sample size and subtract 1.

situation	Typical degrees of freedom	intuition
One-sample variance or t-test	n−1	Estimate one mean
T-test with pooled variance of two independent samples	n₁+n₂−2	Estimate one mean for each group
Chi-square independence test for contingency tables	(r−1)(c−1)	Row sum and column sum create constraints
One-way ANOVA between groups	k−1	Compare the means of k groups
simple regression error	n−2	Estimate two parameters: intercept and slope