Editor’s Message: The Sixth “M”?

In this issue, a special interest paper by Quiben et al1 presents the “Geriatric 5Ms” (Mind, Mobility, Multi-complexity, Medications, and Matters Most) as these concepts relate to and guide physical therapy (PT) practice decisions in the care of older adults. This prioritized approach is subsequently integrated into the new Movement Framework for Older Adults, a second special interest paper in this issue from Leach et al2; both of these articles focus on optimizing patient care. A third paper by Blood et al3 presents the revised education competencies (ECs) and describes how each EC links to the 23 domains of competence, the Geriatric 5Ms, and CAPTE standards. Their report focuses on optimizing geriatric content in PT and PTA education programs. With the 5Ms in patient care and education categories so effectively covered, I asked “Where do the 5Ms fit in the research sphere?”. Certainly, the 5Ms could help guide selection of research topics and questions and remind investigators to include variables that capture these critical concepts. Yet what really stood out to me was the fact that the 5Ms is a multivariate model in which these 5 aspects of aging are interrelated and exert influence on each other. For example, an older adult with pain and depression (Mind) might receive Medications for those conditions that improve function but impair balance and increase fall risk (Mobility). In addition, even within a single “M” aspect, there are multiple components, for example, Mobility involves motivation, balance, ambulation, strength, endurance, etc. For this reason, I would like to suggest a 6th “M” (Multivariate) to consider in the realm of geriatric rehabilitation research. A substantial positive shift in rehabilitation research study designs has occurred over the last few decades: the field has moved beyond univariate designs that answer relatively more straightforward, simple questions to multivariate designs that can answer more complex (and relevant to real life) questions. Instead of using a single independent variable (IV) to predict or explain multifaceted outcomes, or a single dependent variable (DV) to capture the effects of multidimensional interventions, the adoption of multivariable (>1 IV) and multivariate (>1-2 DVs) designs that much more accurately reflect the dynamic complexities of aging, health care, and rehabilitation interventions is now widespread. These multivariate research designs have markedly increased the potential for studies to inform the clinical care of older adults with multidimensional problems.4 The statistical analysis methods appropriate for such multivariate designs ought also to be multivariate, that is, have the ability to simultaneously assess all variables together to detect relationships and identify patterns, correlations, and structures. Studies show that multivariate analysis methods are generally superior at controlling error rates when the number of DVs is higher.5 Yet based on the submissions we receive, and the analyses frequently reported in other studies, the more common statistical analysis approach in rehabilitation research using multivariate designs is univariate (eg, numerous paired t-tests or univariate ANOVAs). This is problematic for several reasons. First, the sample size needed to obtain adequate statistical power (minimum .80) for a univariate t test is small compared to the sample size needed for a multivariate F test, eg, per G*Power6 for 2 groups tested at 2 times, 34 versus 128 participants, respectively. Investigators who calculate their needed sample size a priori using a univariate t test but who will be analyzing multivariate data that would be better explained by a multivariate F test will find themselves severely underpowered for an F test. This too-small sample size and lack of statistical power make it difficult to know if any nonsignificant result is truly nonsignificant, or if the result is actually significant but went undetected (false negative; Type II error). Subsequently, these investigators often run multiple t tests with a stated, uncorrected alpha level of .05. This is problematic because the chance of a false positive finding (the result is actually not significant despite the test saying it was significant; Type I error) is .05 (5%) per comparison.7 When multiple univariate comparisons are conducted, the chances of at least 1 false positive (Family-wise error rate, FWER) rise dramatically; with 5 DVs, it is 20%, and with 12 DVs, it is 54%. Further, it is possible that more than 1 false positive finding could be reported (Per Family error rate, PFER). In these cases, it is difficult to trust that any significant result is truly significant because of the high probability that the result has occurred just by chance. To minimize this chance of error with multiple univariate comparisons, the alpha level must initially be set lower using either a multivariate statistical approach such as MANOVA protection via Tukey’s or Scheffe, or Bonferroni correction following univariate tests.8 For example, with 12 comparisons, the Bonferroni corrected alpha level would be P = .05/12 = .0042 for each comparison. Such univariate correction helps control error rates but is inferior to multivariate analysis approaches to explain overlapping variance or correlations between related variables as a multivariate test would. This ability gives multivariate analysis an increased sensitivity to detect treatment effects. There may be several reasons why univariate analyses are used in studies with multivariate designs. Difficulty obtaining a sufficiently large sample size to support traditional multivariate analyses is certainly a major one. Perhaps the investigator is not familiar with or confident in how to perform and interpret multivariate statistical tests. Or the ease of interpretation of univariate t tests is appealing. Yet experts recommend that multivariate statistical tests be used with multivariate designs.9 What should investigators do if they plan a multivariate design study but anticipate difficulty in obtaining a sufficiently large sample size to permit a traditional multivariate analysis? The best course of action is to collect more data to increase the sample size, but this may not be possible. Alternatively, investigators can (1) consider altering the study design if they predict that the sample size will be small or (2) choose less traditional multivariate statistical analysis approaches that remain accurate with small sample sizes. The latter option may be the only choice if the low number of participants was unexpected (this occurred frequently due to the COVID shutdown). When it is not going to be feasible to recruit or study enough participants to support traditional multivariate analyses, planned modifications in study design should aim to increase the size of the differences (effect size) found over time or between groups to make the differences easier to find with lower power/smaller sample sizes.10 Investigators can reduce variability in several ways: (1) use a within-subjects design when appropriate, (2) add covariates to between-group designs to minimize any differences other than the intervention, (3) choose highly reliable measures without ceiling or floor effects (that are sensitive to change over time if repeated measures are used), (4) choose tests that measure precisely what the intervention aims to change, (5) choose tests that are matched in difficulty level to the ability level of the participants (not too easy, not too difficult), (6) recruit a more homogeneous study sample, and (7) add measurement times [eg, multiple baseline (pre) test sessions and longitudinal follow up (post) tests]. For intervention studies, investigators can aim to enhance the treatment effect by (1) using an intervention that specifically targets the mechanism(s) expected to change, (2) increasing intensity or dose while ensuring that the control or comparison group does not inadvertently receive the same or similar intervention, (3) progressing the difficulty level of the intervention as participants improve, and (4) ensuring the expert delivery of the intervention. When difficulties acquiring a sufficient number of participants are anticipated, increased attention should be given to participant retention, attendance, adherence, and prevention of attrition. Researchers should avoid missing data when possible and consider data imputation methods to avoid case loss. If the small sample size was not anticipated in advance, and traditional multivariate analyses cannot be supported with adequate statistical power, adjustment of the approach to statistical analysis should be made. Current recommendations suggest multivariate methods that are not based on null hypothesis testing.8 Multivariate approaches that are based on “dimension reduction” can work well with a large number of variables even if they are highly correlated and the sample size is small. These include principal components analysis (PCA), redundancy analysis (RDA), and partial least squares – discriminant analysis (PLS-DA), with the latter offering the strongest performance.9 All of these analyses help to answer questions about patterns and sources of variation in a data set, how much of the variance in the outcome variables can be explained by the predictor variables, and which variables are best at separating participants into groups. The use of Bayesian statistical analyses, which work very well with small sample sizes, is increasingly seen in health care research.11 Instead of saying a treatment is simply “effective” or “ineffective,” Bayesian tests provide a probability distribution of the treatment’s effectiveness. These tests provide direct information about the probability of an outcome based on the data provided, for example, “What is the probability that Treatment A will result in independent ambulation within 4 weeks?” Bayesian methods allow the use of prior evidence in addition to the current study data to improve probability estimations and can update estimated probabilities as new data are obtained.12 Wider adoption of Bayesian approaches is limited by the need for advanced statistical expertise, higher computational power, and resources for data management, as well as low familiarity among investigators and grant funders. But the ability to answer clinical questions confidently despite small sample sizes is quite compelling, and geriatric rehabilitation researchers should explore how they can collaborate with Bayesian biostatisticians to strengthen the validity and direct application of their “small N” research findings. I hope that in the future, statistical software for Bayesian analyses will be as accessible and user-friendly as current programs for MANOVA are. Training for researchers in Bayesian analysis approaches could begin now, both in graduate programs and at professional conferences. Small sample sizes are a large barrier to obtaining valid results with traditional multivariate analysis approaches. To avoid compromising the validity of results by using univariate methods with multivariate designs, an additional shift to using multivariate analysis methods that are robust with small sample sizes (including but not limited to Bayesian analyses) is due. Lastly, what not to do when you have a smaller-than-anticipated sample size is to change the title and call it a pilot study. While pilot studies often do have small sample sizes, this characteristic is not what defines a pilot study. The focus of a pilot study is on the conduct of the study (eg, recruitment, resources, study procedures, etc.), rather than on the effectiveness of the intervention.13