Multiple Contrast Tests for Count Data: Small Sample Approximations and Their Limitations

Although count data are collected in many experiments, their analysis remains challenging, especially in small sample sizes. Until now, linear or generalized linear models in Poisson or Negative Binomial distributional families have often been used. However, these data frequently show signs of over-, underdispersion, or even zero-inflation, casting doubt on these distributional assumptions and leading to inaccurate test results. Since their distributions are usually skewed, data transformations (e.g., log-transformation) are not unusual. This underscores the need for statistical methods not to hinge on specific distributional assumptions. We delve into multiple contrast tests that allow general contrasts (e.g., many-to-one or all-pairs comparisons) to analyze count data in multi-arm trials. The methods vary in their effect and variance estimation, as well as in approximating the joint distribution of multiple test statistics, including frequently used methods such as linear and generalized linear models, and data transformations. An extensive simulation study demonstrates that a resampling version effectively controls the Type I error rate in various situations, while also highlighting the method's limitations, including overly liberal Type I error rates. Some standard methods, which have inflated Type I error rates, further underscore the need for alternative approaches. Real data applications further emphasize the applicability of these methods.