Which statistical test to use
Answer a few questions about your data. You get the appropriate test, why, which assumptions it needs and how to check them, plus the R and Python call.
Only needed if you don't know whether the data is normal: from 30 up, the sample mean is approximately normal.
The recommended test
Mann-Whitney U (Wilcoxon rank-sum) test
Compares two independent groups on ranks: tells whether one tends to have higher values.
Why this test
- You don't know whether the data is normal and the sample is small, or you gave no sample size: you cannot lean on the central limit theorem, so I pick a rank-based test, valid either way.
- You have two groups or two measurements.
Assumptions and how to check them
| Assumption | How to check it | Status |
|---|---|---|
| Observations are independent: no subject appears twice and there is no hidden clustering (classes, families, sites). | It is not checked with a test: it follows from the study design. If the data is clustered, mixed models are needed. | To verify |
| To read the result as a difference in medians, the groups must have distributions of the same shape. Otherwise the test only says that one group tends to exceed the other. | Compare the groups' histograms or box plots. | To verify |
| The data has at least an order (it can be ordinal or numeric). | Check that there is a sensible order among the values. | To verify |
If one premise changes
- If you verify that the data (or the differences) is approximately normal: use Independent-samples Welch t test.
What to report alongside the p-value
Rank-biserial correlation, or probability of superiority: how often a value in one group exceeds one in the other. 0 is no effect, plus or minus 1 is complete separation.
How to run it
wilcox.test(y ~ group, data = df, conf.int = TRUE)from scipy import stats
stats.mannwhitneyu(a, b, alternative='two-sided')Test source: Mann & Whitney (1947)
Good to know
- Choosing the test from a preliminary normality test has a cost: it distorts the type I error (Rochon, Gondan and Kieser, 2012). Look at the Q-Q plot and use what you know about the phenomenon.
Before collecting data
This tool picks the test: how much data you need is decided beforehand, with a power calculation.
What this tool does not do
- It picks the test, it does not run it: it never sees your data and produces no p-value.
- When you answer «I don't know» you get the choice that holds either way, not the most powerful one. Checking the premise may earn you a better test.
- It covers the most common designs: one factor, or two variables. It does not cover several factors, covariates, nested or hierarchical data, survival or time series: those need regression or mixed models.
- The threshold of 30 observations is a rule of thumb, not a theorem: with very skewed distributions or outliers it may not be enough.
- The test should be chosen before looking at the results. Trying several and keeping the significant one is a form of p-hacking.
Come utilizzare Which statistical test to use
State what you want to do
Choose whether to compare groups, measure the association between two variables, or compare a sample with a known value.
Describe the data
Give the measurement scale, how many groups you have and whether they are paired. For premises you don't know answer «I don't know».
Read test, assumptions and code
You get the test, the reason, the assumptions with ways to check them, and the R and Python call.
Suggerimenti
- Always report the effect size with its confidence interval: a small p-value with an irrelevant effect is common with lots of data.
- Choose the test before seeing the results and write it in the analysis plan.
Domande frequenti
Student or Welch: which should I use?
Welch, unless you have verified that the variances are equal. Student assumes equal variances and becomes unreliable if they are not, especially with unequal group sizes. Welch assumes nothing about the variances and, when they are equal, loses very little power (Delacre, Lakens and Leys, 2017). In R, t.test also uses Welch by default.
Do I need a normality test before choosing?
It is not the only way and it has costs. With little data the test has low power and misses large departures; with lots of data it rejects irrelevant ones. The Q-Q plot and your knowledge of the phenomenon tell you more. With at least 30 observations per group the mean is approximately normal even for skewed data.
When do I move from chi-square to Fisher's exact test?
When the expected counts (not the observed ones) are below 5 in too many cells. Cochran's (1954) criterion asks that all be at least 5, or that at most 20 per cent be below 5 and none below 1. Fisher's test is always valid, but for large tables it can be slow.
What is the difference between Pearson and Spearman?
Pearson measures the linear relationship between values and is affected by outliers. Spearman applies Pearson to the ranks: it measures the monotonic relationship, resists outliers and also works for ordinal data. If the relationship is a U-shaped curve, neither sees it: look at the plot.
Why aren't three groups compared with three t tests?
With three groups there are three pairwise comparisons and each carries its own 5 per cent type I error: the probability of at least one false positive rises to about 14 per cent. A global test is used (ANOVA, Kruskal-Wallis) and, only if it is significant, pairwise comparisons corrected for multiple comparisons.
Want to learn more?
Read our in-depth articles on related topics: