Use a t-test to compare two groups and ANOVA to compare three or more, then pick the version that matches whether the groups are independent or paired. For two independent groups, make Welch's t-test the default; for three or more independent groups with unequal variances, use Welch's ANOVA; when the data are clearly skewed or contain extreme values, switch to a rank-based test (Mann–Whitney, Wilcoxon signed-rank or Kruskal–Wallis). Whatever test you run, report an effect size and a 95% confidence interval alongside the p-value.
The sections below follow the order of the decision: the design first, then the condition of the data, then what to report.
Start with the experimental design
The design sets the family of tests, and you can settle it before any data are collected. Three points decide it.
How many groups
Two groups call for the t-test family; three or more call for the ANOVA family. Run on two groups, a one-way ANOVA reaches the same conclusion as Student's t-test (F equals t squared), so the two are the same method at different numbers of groups.
Independent or paired
Independent samples means each group is made up of different individuals, such as separate mice in the control and treated groups.
Paired samples means two values come from the same individual or the same matched unit: one mouse before and after dosing, cells from one donor split between two treatments, or wells handled as matched pairs within a batch.
Paired samples need a paired t-test, which tests the within-pair differences. If you analyze paired data with an independent-samples test, variation between individuals is counted as error and power drops. If you analyze independent data with a paired test, the calculation relies on a correspondence that does not exist.
n is the number of independent experimental units
Running one sample three times on the instrument gives three technical replicates, and n is still 1. Average the technical replicates into one value, then use the independent biological replicates as n. Counting technical replicates as n understates the variation and makes p-values too small.
One factor or two
When the experiment has two factors at once (genotype and drug, for example), use a two-way ANOVA so that the interaction between the factors can be tested. Treating the four combinations as four groups in a one-way ANOVA cannot answer whether the drug effect depends on genotype. The decision table below covers single-factor designs only.
One-sided or two-sided
Use a two-sided test by default. A one-sided test fits only when the direction was fixed before data collection and a result in the opposite direction would have no meaning. Switching to a one-sided test after seeing the data halves the p-value and raises the false-positive rate.
Then check the condition of the data
The design sets the family. The condition of the data sets which version within that family to use.
| Design | Data condition | Test |
|---|---|---|
| Two groups, independent | Each group roughly normal | Welch's t-test (default) |
| Two groups, independent | Each group roughly normal, with grounds to treat the variances as equal | Student's t-test |
| Two groups, independent | Clearly skewed, extreme values, or ordinal data | Mann–Whitney U test |
| Two groups, paired | Differences roughly normal | Paired t-test |
| Two groups, paired | Differences clearly skewed or with extreme values | Wilcoxon signed-rank test |
| Three or more groups, independent | Residuals roughly normal, equal variances | One-way ANOVA, with Tukey HSD as the post hoc test |
| Three or more groups, independent | Residuals roughly normal, unequal variances | Welch's ANOVA, with Games–Howell as the post hoc test |
| Three or more groups, independent | Clearly skewed, extreme values, or ordinal data | Kruskal–Wallis test, with Dunn's test as the post hoc test |
| Three or more groups, repeated measures on the same individuals | Differences roughly normal | Repeated-measures ANOVA; Friedman test when the condition is not met |
What normality refers to
The t-test and ANOVA assume that the values within each group (the residuals) come from a normal distribution. The paired t-test assumes this of the differences. A histogram of all groups pooled together shows the between-group difference mixed with the within-group spread, so it cannot be used to judge the assumption.
With 3 to 6 samples per group, normality tests such as Shapiro–Wilk have very low power, and a non-significant result only means no evidence of departure was seen. A more dependable approach is to plot every data point and to draw on what earlier measurements of the same kind looked like. Right-skewed data such as ratios, fold changes and concentrations are often log-transformed before testing.
Handling extreme values
When you see an extreme value, trace its source first: a pipetting error, an instrument fault or a recording error can be confirmed from the lab record. A value with a clear technical cause can be excluded, with the reason stated in the methods. Without a technical cause, the value is part of the data; you can switch to a rank-based test, or report the result both with and without the value. Set the exclusion rule before the analysis.
Student versus Welch
Student's t-test pools the two group variances into one estimate, which requires the population variances to be equal. Welch's t-test uses each group's variance separately and adjusts the degrees of freedom from the data (Welch, 1947), which is why its degrees of freedom are usually not a whole number.
When the two variances are equal, Welch and Student give very similar results. When the variances differ and the group sizes differ too, the type I error rate of Student's t-test moves away from the nominal 5%, while Welch's stays close to it. Ruxton (2006) and Delacre et al. (2017) therefore both recommend Welch's t-test as the default for two independent groups, without first running a test of equal variances to choose between the two. Treatments often change the spread as well as the mean, so unequal variances are common in experimental data.
Rank-based tests
The Mann–Whitney U test ranks all values together and compares the ranks of the two groups. The Wilcoxon signed-rank test is its counterpart for paired samples, and the Kruskal–Wallis test (Kruskal & Wallis, 1952) covers three or more groups. These tests do not require normality and are less affected by extreme values.
Two limits are worth knowing. First, with very few samples a rank test has a floor on the p-value it can return. With 3 samples per group, the smallest two-sided p-value from an exact Mann–Whitney test is 0.10, so it cannot fall below 0.05 however large the difference is. Second, the Mann–Whitney test asks whether values in one group tend to be larger than values in the other; it can be read as a comparison of medians only when the two distributions have the same shape. It is also affected when the two groups differ in spread (Ruxton, 2006).
Three or more groups: ANOVA and multiple comparisons
Repeated t-tests raise the false-positive rate
Each test judged at α = 0.05 has a 5% chance of a significant result when no real difference exists. The more comparisons you make, the higher the chance of at least one false positive. If the comparisons are independent, that chance is 1 − 0.95ᵏ, where k is the number of comparisons.
| Groups | Pairwise comparisons | Chance of at least one false positive |
|---|---|---|
| 3 | 3 | 14.3% |
| 4 | 6 | 26.5% |
| 5 | 10 | 40.1% |
Real pairwise comparisons share the same data and are not fully independent, so the exact figures differ slightly; the direction is the same.
Post hoc tests after ANOVA
A one-way ANOVA uses a single F-test to ask whether all group means are equal, which holds the overall type I error rate at 5%. A significant F-test says only that at least one group differs. Finding out which groups differ takes a post hoc test.
- All pairwise comparisons: Tukey HSD.
- Each group against the control only: Dunnett's test.
- Unequal variances: Welch's ANOVA with Games–Howell.
- After Kruskal–Wallis: Dunn's test with adjusted p-values.
If only a few comparisons chosen in advance matter, you can run just those tests and adjust them with the Bonferroni or Holm method. Decide which groups to compare before you see the data.
Worked example: two independent groups
The numbers below are made up to illustrate the calculation. They are not real experimental data. Suppose qPCR was used to measure the relative expression of a gene, with 6 independently cultured samples in each of the control and treated groups.
| Group | Values | Mean | SD |
|---|---|---|---|
| Control | 1.02, 0.95, 1.10, 0.98, 1.05, 0.90 | 1.00 | 0.072 |
| Treated | 1.35, 1.80, 1.12, 2.05, 1.52, 1.28 | 1.52 | 0.348 |
Design. Two groups, each made up of different cultures, so the samples are independent and the t-test family applies.
Data condition. The Shapiro–Wilk test gives W = 0.995, p = 0.997 for the control group and W = 0.951, p = 0.747 for the treated group, so there is no evidence of departure from normality; at n = 6 this test has limited power, and the spread of the points still needs a look. The treated SD is 4.8 times the control SD, a variance ratio of 23.5. A median-centered Levene test gives F(1, 10) = 6.26, p = 0.031. The two groups clearly differ in spread, so Welch's t-test is used.
Test. The difference in means is 0.52 with a standard error of 0.145, giving t = 3.58 on 5.43 degrees of freedom and a two-sided p = 0.014. The 95% confidence interval for the difference is 0.16 to 0.88. The degrees of freedom of Welch's t-test fall between n − 1 for the smaller group and n₁ + n₂ − 2 for both groups combined, here 5 to 10; the treated variance is far larger than the control variance, so the value lands close to 5.
Effect size. The pooled SD is 0.251, so Cohen's d = 2.07; with the small-sample correction, Hedges' g = 1.91. When the two SDs differ this much, the pooled SD is a compromise scale, so report each group's own SD as well.
Comparison with other tests. Student's t-test on the same data also gives t = 3.58 (the two t values are identical when the group sizes are equal), but on 10 degrees of freedom, with p = 0.005 and a confidence interval of 0.20 to 0.84. Student's t-test assumes equal variances, which these data do not meet, so its interval is too narrow. The Mann–Whitney test gives U = 36 with an exact p = 0.0022, because all 6 treated values are larger than every control value; that p-value carries no information about the size of the difference.
How to report it. Relative expression was higher in the treated group than in the control group (means 1.52 vs 1.00; difference 0.52, 95% CI 0.16 to 0.88; Welch's t(5.43) = 3.58, p = 0.014; Hedges' g = 1.91).
Report effect size and confidence interval
The numbers to give besides p
A p-value states how probable data as extreme as yours, or more extreme, would be if the null hypothesis were true. The American Statistical Association's 2016 statement points out that a p-value does not measure the size of an effect or the importance of a result, and that scientific conclusions should not rest only on whether a p-value passes a threshold (Wasserstein & Lazar, 2016). With a large enough sample, a very small difference can produce a very small p-value; with a small sample, a difference worth attention may not reach significance.
A complete result includes at least the following.
- n for each group, and what unit n counts (animals, donors, independent experiments).
- Mean and SD for each group; for skewed data, the median and interquartile range.
- The name and version of the test (for example Welch or Student, one-sided or two-sided).
- The test statistic, degrees of freedom and exact p-value.
- An effect size and its 95% confidence interval. For two groups, the mean difference with Cohen's d or Hedges' g is common; for ANOVA, η² or ω² (Lakens, 2013).
- The method used to adjust for multiple comparisons.
Reading the effect size and the confidence interval
Cohen's d and Hedges' g express the mean difference in SD units, so d = 2 means the group means are two SDs apart. The labels small, medium and large for 0.2, 0.5 and 0.8 are conventions only; whether an effect matters depends on the measurement and the question in that field (Lakens, 2013).
The confidence interval shows how precise the estimate is. The interval in the worked example runs from 0.16 to 0.88, so anything from a modest difference to a difference nearly as large as the control mean is compatible with these data; with 6 samples per group the estimate is not precise. By the same reasoning, a non-significant result cannot be read as "the two groups do not differ". A wide interval that includes 0 means the data are not enough to conclude either way.
Check your choice with the tools
Lab Grimoire's automatic test selection tool picks a test from the structure of the data (how many groups, paired or not). If you already know the method, go straight to the t-test tool or the ANOVA tool. Before testing, use the descriptive statistics tool to look at each group's mean, SD and distribution. These tools compute in the browser with R (webR), show the results of the assumption checks, and give R code you can rerun on your own computer. The tools do the arithmetic. Whether the groups are independent and what unit n counts remain judgments for someone who knows the experimental design.
References
- Welch BL. The generalization of ‘Student's’ problem when several different population variances are involved. Biometrika. 1947;34(1–2):28–35. doi:10.1093/biomet/34.1-2.28
- Kruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. J Am Stat Assoc. 1952;47(260):583–621. doi:10.1080/01621459.1952.10483441
- Ruxton GD. The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test. Behav Ecol. 2006;17(4):688–690. doi:10.1093/beheco/ark016
- Delacre M, Lakens D, Leys C. Why psychologists should by default use Welch's t-test instead of Student's t-test. Int Rev Soc Psychol. 2017;30(1):92–101. doi:10.5334/irsp.82
- Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863
- Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129–133. doi:10.1080/00031305.2016.1154108