Sampling and Sampling Distributions | Foundations of Hypothesis Testing

Sampling is the process of selecting some observation units from a population to create a sample, and inferring the population from that information. What you actually observe is a sample, but what you want to know about is usually a population that has not yet been completely observed.

Population, Sample, Parameter, and Statistic#

The population is the entire unit being studied, and the sample is the part that is actually selected and observed. The value that represents a population is called a population, and the value calculated from a sample is called a statistic. Statistical inference involves estimating or testing unknown parameters from statistics.

level	subject	Typical value	nature
population	All units to be studied	mean μ, variance σ², ratio p	Usually unknown and estimated through inference
sample	n units selected from the population	Mean X̄, variance s², ratio p̂	Estimate the population by calculating from observed data

Population and sample size

N=\text{population size},\qquad n=\text{sample size}

sample mean

\bar X=\frac{1}{n}\sum_{i=1}^{n}X_i

Symbol	Meaning
N	total number of units in the population
n	number of sample units
Xᵢ	the ith observation in the sample
μ、σ²	Population mean and variance
X̄、s²	sample mean and sample variance
p、p̂	Population proportion and sample proportion

Why is sampling necessary?#

Population is too large or cannot be enumerated in its entirety
Save time, manpower, and costs
Destructive testing cannot measure all units.
Small-scale research allows for enhanced training, measurement, and quality control.
With probability sampling, the uncertainty caused by sampling can be statistically evaluated.

Sampling error and sampling bias#

Repeated sampling from the same population will naturally cause the statistics to fluctuate. This is sampling error. On the other hand, the systematic deviation of the sample from the population because certain individuals are more likely or less likely to be selected is called sampling bias.

problem	cause	Will it improve with increasing sample size?
sampling error	Natural variation that randomly samples different individuals	usually smaller
sampling bias	Systematic distortions due to sampling frame, inclusion method, and nonresponse	It won't go away just by making it bigger

Main probability sampling methods#

1. Simple random sampling#

A method in which individuals are randomly selected from a sampling frame, and each possible sample of size n has the same selection opportunity. Although the concept is straightforward, a complete sampling frame is required.

2. Systematic Sampling#

Randomly choose a starting point from the first k units, then sample every k units thereafter. While easy to implement, bias will occur if the roster has the same frequency as the sampling interval.

k\approx\frac{N}{n}

3. Stratified sampling#

Divide the population into non-overlapping strata based on important characteristics and randomly sample from each stratum. Ensures inclusion of important subpopulations. When sampled disproportionately, weighting is required for analysis of the entire population.

4. Cluster Sampling#

Create a natural group such as a hospital, school, or community, and select a portion of it at random. There are single-stage methods that look at everyone in a selected cluster, and multi-stage methods that extract more individuals.

method	Sampling procedure	Main benefits	main danger
simple random sampling	Select units from the complete population list	Concept and analysis are direct	Difficult to obtain complete roster
systematic sampling	every k units from a random starting point	easy to implement	Roster periodicity
stratified sampling	Sample from every stratum	Representativeness of subpopulations	Non-proportional sampling requires weights
cluster sampling	Select some natural groups or clusters	Reduce geographic and implementation costs	Similarity within clusters reduces the amount of effective information

Central Limit Theorem#

When the observed values are independently and identically distributed and the population has a finite mean μ and variance σ², as n increases, the distribution that standardizes the sample mean approaches the standard normal distribution.

Approximate distribution of sample mean

\bar X\approx N\!\left(\mu,\frac{\sigma^2}{n}\right)

standardized shape

\frac{\bar X-\mu}{\sigma/\sqrt n}\xrightarrow{d}N(0,1)\qquad(n\to\infty)

If the population is normally distributed, the sample mean is also normally distributed, regardless of n. If it's non-normal, you'll need a large enough n, but n≥30 is just a rule of thumb, and the more skewed, thick tails, and outliers you have, the bigger the sample you need.

From One Sample to All Possible Samples#

If we simply randomly sample n units from a finite population of N units without considering their order and without reconstruction, the possible number of samples is C(N,n), and one sample mean can be calculated from each sample.

possible sample size

m=\binom{N}{n}

mean of jth sample

\bar X_j=\frac{1}{n}\sum_{i=1}^{n}X_{ji},\qquad j=1,2,\ldots,m

What Is a Sampling Distribution?#

The probability distribution of a statistic over all possible samples, or conceptual repetitions of sampling, is its sampling distribution. The distribution of the repeatedly obtained X̄ values is the sampling distribution of the sample mean.

distribution	Contained values	question to answer
population distribution	each observation in the population	What is the shape of the entire population?
sample data distribution	n observations of one sample	What is the shape of these data?
sampling distribution of statistics	X̄, p̂, etc. across repeated samples	How do the results change if the study is repeated?

Sampling Distribution of the Sample Mean#

The sample mean is an unbiased estimator if X₁,…,Xₙ are independent observations from the same population with mean μ and variance σ². The mean of the repeated sample means is equal to the population mean.

center of sampling distribution

E(\bar X)=\mu

variance of sampling distribution

\operatorname{Var}(\bar X)=\frac{\sigma^2}{n}

standard error of the sample mean

\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}

Supplement 1: Linear combination of random variables

Uppercase Xᵢ is a random variable whose value is uncertain before sampling, and lowercase xᵢ is a fixed value observed after sampling. The formula for deriving the sampling distribution deals with Xᵢ, which changes with repeated sampling.

L=c_1X_1+\cdots+c_nX_n=\sum_{i=1}^{n}c_iX_i

Expected value

E(L)=\sum_{i=1}^{n}c_iE(X_i)

Variance in the independent case

\operatorname{Var}(L)=\sum_{i=1}^{n}c_i^2\operatorname{Var}(X_i)

If there is a correlation

\operatorname{Var}(L)=\sum_{i=1}^{n}c_i^2\operatorname{Var}(X_i)+2\sum_{i<j}c_ic_j\operatorname{Cov}(X_i,X_j)

Supplement 2: Derivation of mean and standard error of sample mean

Sampling Without Replacement from a Finite Population

If we enumerate all C(N,n) samples, each population value appears in C(N−1,n−1) samples. From this combinatorial count, we get E(X̄)=μ. A pair of two values appears in C(N−2,n−2) samples, and arranging the square and cross terms yields a finite population correction.

impartiality

E(\bar X)=\mu

Variance in a finite population

\operatorname{Var}(\bar X)=\frac{N-n}{n(N-1)}\sigma^2

Standard error in a finite population

\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}\sqrt{\frac{N-n}{N-1}}

Derivation by linear combination of independent random variables

If N is sufficiently larger than n, each draw can be treated as approximately independent, and X̄ can be treated as a linear combination of coefficients 1/n.

sample mean

\bar X=\sum_{i=1}^{n}\frac{X_i}{n}

average

E(\bar X)=\sum_{i=1}^{n}\frac{1}{n}E(X_i)=\mu

dispersion

\operatorname{Var}(\bar X)=\sum_{i=1}^{n}\frac{1}{n^2}\operatorname{Var}(X_i)=\frac{\sigma^2}{n}

standard error

\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}

Finite population methods accurately preserve dependencies due to nonrecovery sampling. The linear combination method is a simple approximation where N≫n and ignores the influence of a finite population.

If σ is unknown, estimate the standard error using s/√n. Sampling without replacement is performed from a finite population, and if the sampling rate n/N cannot be ignored, a finite population correction is applied.

Estimate by sample standard deviation

\widehat{\operatorname{SE}}(\bar X)=\frac{s}{\sqrt n}

Finite population correction

\operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}\sqrt{\frac{N-n}{N-1}}

Why is the sample variance divided by n-1?#

Since X̄ is estimated from the same data, the sum of n deviations Xᵢ−X̄ is 0, and n−1 can vary independently. When divided by n−1, the sample variance becomes an unbiased estimator of the population variance.

sample variance

s^2=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar X)^2

sample standard deviation

s=\sqrt{s^2}

impartiality

E(s^2)=\sigma^2

Supplement 3: Estimation of population variance using sample variance

Averages the within-sample variation v in the denominator n over all possible samples of a finite population. We use the fact that each xᵢ² appears C(N−1,n−1) times and each xᵢxⱼ appears C(N−2,n−2) times.

Within-sample variation of denominator n

v=\frac{1}{n}\sum(X_i-\bar X)^2

mean over possible samples

E(v)=\frac{n-1}{n}\frac{N}{N-1}\sigma^2

Correction for large populations

s^2=\frac{n}{n-1}v=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar X)^2

How does sample size affect standard error?#

Since the standard error is inversely proportional to √n, increasing the sample size concentrates the distribution of the sample mean, but the improvement is not linear. To halve the standard error, quadruple the sample size.

\operatorname{SE}(\bar X)\propto\frac{1}{\sqrt n}