Exploring the Effects of Sampling Variability, Scale Variability, and Node Aggregation on the Consistency of Estimated Networks

Work surrounding the replicability and generalizability of network models has increased in recent years, prompting debate on whether network properties can be expected to be consistent across samples. To date, certain methodological practices may have contributed to observed inconsistencies, including use of single-item indicators and non-identical measurement tools. The current study used a resampling approach to disentangle the effects of sampling variability from scale variability when assessing network replicability in empirical data. Additionally, we explored whether consistencies in network characteristics were improved when more items were aggregated to estimate node scores, which we hypothesized should yield more representative measures of latent constructs. Overall, using different scales produced more variability in network properties than using different samples, but these discrepancies were markedly reduced with larger samples and greater node aggregation. Findings underscored the impact of aggregating items when estimating nodes: Multi-item indicators led to denser networks, higher network sensitivity, greater estimates of global strength, and greater levels of consistency in network properties (e.g., edge weights, centrality scores). Taken together, variability in network properties across samples may arise from poor measurement conditions; additionally, variability may reflect properties of the true network model and/or the measurement instrument. All data and syntax are openly available online (https://osf.io/m37q2/).