Inflation of Familywise Error Rate in Treatment Efficacy Testing Due to the Reallocation of Significance Levels Based on Safety Data

In randomized clinical trials comparing two standard treatments, a two-sided test for efficacy at a significance level α is typically used when neither treatment is expected to be superior at the design stage. This two-sided test comprises two one-sided tests, each conducted at a significance level of α/2. If safety data later suggest that one treatment is not clinically acceptable due to a higher rate of adverse events, investigators may reallocate the α/2 significance level originally assigned to the one-sided efficacy test for that treatment to the other one-sided test. This results in a two-stage procedure. We examine the impact of such reallocation on the familywise error rate (FWER). Using theoretical derivations and simulation studies, we show that FWER can exceed the nominal level α when the treatment with fewer adverse events tends to show greater efficacy. Therefore, the two-stage procedure should be avoided when strict control of FWER is a priority. These findings emphasize the need for caution when reallocating significance levels based on auxiliary information and have implications beyond clinical trials, particularly in adaptive statistical methodologies.