Common practice in analyzing paired rating scale data: A simulation study exploring the appropriateness of the Wilcoxon Matched Pairs Test

The Wilcoxon matched pairs signed ranks (WMPSR) test is one of the most common statistical tests used to analyze paired ordinal data in medical research. Apart from methodological problems that arise from calculating ranked differences for ordinal data, the analyzed data sets are often highly skewed (e.g. image quality ratings), leading to floor and ceiling effects that produce a large number of tied ranks and may bias the test statistic. An alternative procedure for analyzing paired ordinal data is the sign test, though it is known to be less powerful than the WMPSR test. The aim of our simulation study is to assess the appropriateness of these two tests (both exact and asymptotic) for paired ordinal data by comparing their type I error and statistical power using data simulated to conform to typical situations in a medical context.
Data were simulated by modeling five different distributions and seven different effect sizes for sample sizes ranging from 10 to 100 (10 000 runs were performed for each condition). The results indicate that type I error for the WMPSR test (no matter whether asymptotic or exact) exceeds the nominal type I risk when data are positively skewed (more than 14% for N = 100), while the WMPSR test's power is comparable to that of the asymptotic sign test under all other conditions. The exact tests generally show a lower power than the asymptotic tests, especially for sample sizes below N = 20.
In general, (1) we recommend using asymptotic tests instead of exact tests to analyze paired ordinal data, and (2) the sign test should be generally preferred to the WMPSR test.