Skip to main content
Lab Grimoire
TW EN
Coffee
Lab Methods

After a Significant ANOVA: How to Choose a Post Hoc Test

A significant ANOVA does not show which groups differ. Compare Tukey HSD, Games–Howell, Dunnett, Holm and Dunn's test, with a worked example.

Author
CY
Published
Biomedical Principles and Techniques · Part 2 of 4

A significant one-way ANOVA says only that the group means are not all equal. It does not say which groups differ. To answer that, run a post hoc test: Tukey HSD for all pairwise comparisons when variances are similar, Games–Howell when variances are unequal, and Dunnett when every group is compared with one control. Report an effect size and a confidence interval for each difference alongside the p value.

What the omnibus F test does and does not tell you

One-way ANOVA splits the total variation in the data into two parts: between-group variation (how far each group mean is from the grand mean) and within-group variation (how much values differ inside a group). The F statistic is the between-group mean square divided by the within-group mean square. The larger F is, the harder it is to explain the spread of the group means by random variation within groups.

The test looks at all groups at once, so it is called the omnibus F test. Its null hypothesis is that all population means are equal. Rejecting it tells you that at least one inequality exists. Three things remain unknown:

  • Which two groups differ.
  • The direction and size of the difference.
  • Whether the difference matters in practice.

Why separate t tests for every pair do not work

Three groups give 3 pairwise comparisons, four groups give 6, and five groups give 10. If each comparison uses α = 0.05, the probability of at least one false positive rises with the number of comparisons. For independent comparisons it is about 14.3% for 3 comparisons, 26.5% for 6, and 40.1% for 10 (calculated as 1 − 0.95^m, where m is the number of comparisons). A post hoc test makes several comparisons while holding that probability at the chosen level.

With only two groups, no post hoc test is needed. Use a t test.

Family-wise error rate and false discovery rate

Multiple-comparison corrections control one of two quantities:

  • Family-wise error rate (FWER): the probability of at least one false positive in the whole set of comparisons. Tukey HSD, Dunnett, Bonferroni, and Holm all control the family-wise error rate.
  • False discovery rate (FDR): the expected proportion of false positives among all results declared significant. The procedure published by Benjamini and Hochberg in 1995 controls the false discovery rate.

After an ANOVA there are usually a few to a dozen comparisons, and each conclusion is cited on its own, so the convention is to control the family-wise error rate. The false discovery rate suits screening analyses with hundreds or thousands of comparisons, such as gene expression data: the researcher accepts that a small share of the significant list is false in exchange for more power. The two quantities are defined differently, so state which one you used.

The omnibus test and the post hoc test can disagree

The omnibus F test and a post hoc test are different tests, and their conclusions sometimes differ. The F test can be significant while Tukey HSD finds no significant pair: the F test responds to the overall spread of the group means, and the difference may come from a combination of groups without being concentrated in one pair. The reverse can also happen: a non-significant F test with one pair significant under Tukey HSD.

Tukey HSD and Dunnett control the family-wise error rate at α by themselves, so they do not mathematically require a significant F test. In practice, the common sequence is still the omnibus test first and the post hoc test second. Decide which sequence you will follow before the analysis and write it in the methods section. Do not switch after seeing the results.

How to choose a post hoc test

Three things decide the choice: which groups you want to compare, whether the group variances are similar, and whether means describe the data well.

Answer these three questions in order to land on one row of the table:

  1. Scope of comparison: all pairs, each group against a control, or a few pairs specified in advance.
  2. Variances: whether the group standard deviations are similar. Treat the variances as unequal when the largest and smallest standard deviations differ severalfold, or when the Brown–Forsythe test is significant.
  3. Data type: compare means when each group is roughly symmetric; use a rank-based method for strongly skewed or ordinal data.

When each group is very small, tests of normality and equal variances have low power themselves, and a non-significant result is not proof that an assumption holds. Look at the raw data points as well, and draw on past experience with the same kind of measurement.

Test Compares Assumes Use when
Tukey HSD All pairs of groups Normality, equal variances After a classical one-way ANOVA, when every pair is of interest
Games–Howell All pairs of groups Normality; equal variances not required Variances or sample sizes differ widely; pairs with Welch ANOVA
Dunnett Each treatment group against one control Normality, equal variances Only treatment-versus-control differences matter
Bonferroni / Holm A few comparisons specified in advance No specific test; adjusts p values only A small set of planned comparisons written down before the analysis
Dunn's test All pairs of groups (rank-based) Normality not required After a Kruskal–Wallis test

Tukey HSD

Tukey proposed this method in 1949. It is based on the studentized range distribution, uses the within-group mean square from the ANOVA as a common error estimate, handles all pairwise comparisons at once, and holds the family-wise error rate at α. The version for unequal group sizes is called Tukey–Kramer, and most software applies it automatically. Tukey HSD also gives a confidence interval for each pairwise difference. An interval that excludes 0 means that pair is significant.

Games–Howell

Tukey HSD assumes equal group variances. When that assumption fails and the group sizes also differ, the common error estimate becomes inaccurate. Games–Howell calculates a separate standard error for each pair and adjusts the degrees of freedom in the same way as Welch's test, so it does not need equal variances. When the omnibus test is Welch ANOVA, use Games–Howell as the post hoc test. The two share the same assumptions.

Dunnett

Use Dunnett when the design is one control group plus several treatment groups and only the difference between each treatment and the control matters. With k groups, Tukey HSD makes k(k−1)/2 comparisons and Dunnett makes k−1. Fewer comparisons need a smaller correction, so Dunnett has more power than Tukey HSD for those comparisons on the same data. The cost is that differences between treatment groups are not tested.

Bonferroni and Holm

Bonferroni divides α by the number of comparisons m or, equivalently, multiplies each p value by m. It does not depend on a particular distribution and can be applied to p values from any test, but it becomes conservative as the number of comparisons grows. Holm published a stepwise version in 1979: sort the p values from smallest to largest, multiply the smallest by m, the next by m−1, and so on. Holm also controls the family-wise error rate and is never less powerful than Bonferroni, so choose Holm when deciding between the two. Both methods suit a few comparisons specified in advance. For all pairwise comparisons, Tukey HSD usually has more power.

Dunn's test

When the data depart clearly from normality or are ordinal, use Kruskal–Wallis as the omnibus test and Dunn's test as the post hoc test. Dunn's test ranks all the data together, compares the mean ranks of two groups, and then adjusts the p values for multiple comparisons (commonly with Bonferroni or Holm). It compares ranks, so state the conclusion as "values in one group tend to be higher than in the other," not as a difference in means.

A complete worked example with three groups

The data below are illustrative and invented for this explanation. They are not experimental results. Suppose the relative expression of a protein (arbitrary units) is measured in a control group, a low-dose group, and a high-dose group, with 6 independent samples per group.

Group Raw values Mean SD
Control 4.2, 4.8, 5.1, 4.5, 5.0, 4.6 4.70 0.33
Low dose 5.0, 5.6, 4.9, 5.8, 5.3, 5.5 5.35 0.35
High dose 6.4, 5.9, 6.8, 6.1, 7.0, 6.3 6.42 0.42

You can calculate the group means and standard deviations first with the descriptive statistics tool, which also helps you spot entry errors and outliers.

Step 1: the omnibus F test

The three standard deviations are similar (0.33 to 0.42). Levene's test centered on the median (the Brown–Forsythe test) gives F(2, 15) = 0.10, p = 0.90, so there is no evidence of unequal variances and a classical one-way ANOVA is used.

Source Sum of squares df Mean square
Between groups 9.014 2 4.5072
Within groups 2.043 15 0.1362
Total 11.058 17

F(2, 15) = 4.5072 ÷ 0.1362 = 33.09, p < 0.001 (p = 3.2 × 10⁻⁶). The three group means are not all equal.

Step 2: effect size

η² = between-group sum of squares ÷ total sum of squares = 9.014 ÷ 11.058 = 0.815. ω² = (9.014 − 2 × 0.1362) ÷ (11.058 + 0.1362) = 0.781. The invented data were given clearly separated groups, so the effect size is large. Values from real experiments are usually much smaller.

Step 3: Tukey HSD

All three pairs are of interest and the variances are similar, so Tukey HSD is used.

Comparison Mean difference 95% confidence interval Adjusted p value
Low dose − Control 0.65 0.10 to 1.20 0.021
High dose − Control 1.72 1.16 to 2.27 < 0.001
High dose − Low dose 1.07 0.51 to 1.62 < 0.001

The Tukey HSD threshold can be calculated directly. With three groups, 15 error degrees of freedom, and α = 0.05, the critical value of the studentized range is 3.673, and the threshold is 3.673 × √(0.1362 ÷ 6) = 0.55. Any pair whose means differ by more than 0.55 is significant, and all three differences in the table (0.65, 1.72, 1.07) exceed it.

All three confidence intervals exclude 0, so all three pairs are significant. The low-dose versus control difference is the smallest, and the lower limit of its interval, 0.10, is close to 0. The evidence for this pair is the weakest.

With a Bonferroni correction for the same pair (unadjusted p = 0.0081, multiplied by 3), p = 0.024, slightly higher than the 0.021 from Tukey HSD. With only 3 comparisons the gap is small. As the number of comparisons grows, Bonferroni becomes more conservative relative to Tukey HSD.

If the Brown–Forsythe test had been significant for these data, the sequence would change to Welch ANOVA followed by Games–Howell. For strongly skewed data, it would change to Kruskal–Wallis followed by Dunn's test. Change the omnibus test and the post hoc test as a set. If the research question were only whether each dose differs from the control, Dunnett would be the right choice, with two comparisons.

Effect size and how to write the result

η² and ω²

The p value addresses whether a difference could come from random variation. The effect size addresses how much of the variation the grouping explains. Report both.

  • η² (eta squared): the between-group sum of squares as a proportion of the total sum of squares. It is simple to calculate, but it describes the sample and overestimates the population value. The smaller the sample, the larger the overestimate.
  • ω² (omega squared): subtracts an error term in the formula and is less biased than η². Lakens (2013) sets out how to calculate both and notes that ω² is less biased. With modest sample sizes, report ω², or report both.

η² and ω² describe the overall effect of grouping. Readers usually care more about how far apart two particular groups are, so give the mean difference and 95% confidence interval for each pairwise comparison. These are in the units of the original measurement and are the easiest to interpret.

The sentence that goes in the report

A complete results statement has five parts: the omnibus test used, the F value with both degrees of freedom, the p value, the effect size, and the name of the post hoc test with the result for each pair. For the worked example:

A one-way ANOVA showed that relative expression differed among the three groups, F(2, 15) = 33.09, p < 0.001, η² = 0.82, ω² = 0.78. In Tukey HSD post hoc tests, the high-dose group was higher than the control group (mean difference 1.72, 95% confidence interval 1.16 to 2.27, p < 0.001) and higher than the low-dose group (mean difference 1.07, 95% confidence interval 0.51 to 1.62, p < 0.001), and the low-dose group was higher than the control group (mean difference 0.65, 95% confidence interval 0.10 to 1.20, p = 0.021).

The methods section should also state four things: the sample size per group and what counts as a sample (independent individuals or repeated measurements), which assumptions were checked, the significance level, and the software and version used.

Four common errors

  • Writing only "ANOVA was significant, p < 0.05" with no post hoc test and no statement of which groups differ.
  • Using Welch ANOVA or Kruskal–Wallis as the omnibus test and then Tukey HSD as the post hoc test. The assumptions of the two steps do not match.
  • Deciding after seeing the data to compare only the pairs with the largest differences, then dividing α by that number of pairs for Bonferroni. The scope of comparison must be set before looking at the data, or the correction is too small.
  • Writing "the two groups are the same" for a non-significant result. Non-significance means the evidence is insufficient. Use the confidence interval to show the range of plausible differences.

Run it with the tool

The ANOVA tool does the calculation in the browser. You can select classical one-way ANOVA, Welch ANOVA, or Kruskal–Wallis, or let the tool recommend a method from the data. The pairwise comparisons follow the omnibus test automatically: Tukey HSD with classical ANOVA, Games–Howell with Welch ANOVA, and Dunn's test with Holm adjustment with Kruskal–Wallis. The results list the Shapiro–Wilk test for each group, the Brown–Forsythe test, the effect size, and the adjusted p value for each pair. The tool does not currently include Dunnett's test or false discovery rate correction. Use R or other statistical software when you need them.

If you are not sure which test fits your data, start with the automatic statistical test tool.

References

  • Tukey JW. Comparing individual means in the analysis of variance. Biometrics. 1949;5(2):99–114. doi:10.2307/3001913
  • Games PA, Howell JF. Pairwise multiple comparison procedures with unequal N's and/or variances: a Monte Carlo study. J Educ Stat. 1976;1(2):113–125. doi:10.3102/10769986001002113
  • Dunnett CW. A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc. 1955;50(272):1096–1121. doi:10.1080/01621459.1955.10501294
  • Dunn OJ. Multiple comparisons using rank sums. Technometrics. 1964;6(3):241–252. doi:10.1080/00401706.1964.10490181
  • Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B. 1995;57(1):289–300. doi:10.1111/j.2517-6161.1995.tb02031.x
  • Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863

References

  1. Tukey JW. Comparing individual means in the analysis of variance. Biometrics. 1949;5(2):99–114.
  2. Games PA, Howell JF. Pairwise multiple comparison procedures with unequal N's and/or variances: a Monte Carlo study. Journal of Educational Statistics. 1976;1(2):113–125.
  3. Dunnett CW. A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association. 1955;50(272):1096–1121.
  4. Dunn OJ. Multiple comparisons using rank sums. Technometrics. 1964;6(3):241–252.
  5. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B. 1995;57(1):289–300.
  6. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology. 2013;4:863.

Frequently Asked Questions

Do I always need a post hoc test after a significant ANOVA?

Yes, whenever the research question is which groups differ. The omnibus F test shows only that the group means are not all equal, not which two differ. With only two groups, no post hoc test is needed; use a t test.

Should I choose Tukey HSD or Bonferroni?

Use Tukey HSD to compare all pairs of groups when the group variances are similar; it usually has more power. Use Bonferroni or Holm for a few comparisons specified in advance. Of those two, Holm is never less powerful than Bonferroni.

Which post hoc test should I use when group variances are unequal?

Use Games–Howell. It calculates a separate standard error for each pair and adjusts the degrees of freedom, so it does not require equal variances. Pair it with Welch ANOVA as the omnibus test.

Which post hoc test follows a significant Kruskal–Wallis test?

Use Dunn's test with a Holm or Bonferroni adjustment of the p values. Dunn's test compares ranks, so state that values in one group tend to be higher, not that the means differ.

Found this useful? RSS · X · Buy me a coffee