Bayes Factor vs. Posterior Predictive Model Assessment: Insights from Ordinal Constraints

Abstract A central element of statistical inference is a good model specification where researchers specify models that capture differing theoretical positions. We argue that methods of inference forcing researchers to use models that may not be appropriate for their research question are not as desirable as methods with no such constraints. We ask how posterior predictive model assessment methods such as Watanabe–Akaike information criterion and leave-one-out cross-validation perform when theoretical positions correspond to different space restrictions on a common parameter space. One of the main theoretical relations is nesting—where the parameter space of one model is a subset of that for another. A good example is a general model that admits any set of preferences; a nested model is one that admits only preferences that obey transitivity. We find that posterior predictive methods fail in these cases: More constrained models are not favored even when data are compatible with the constraint. Researchers who use posterior predictive methods are forced to partition the parameter space into non-overlapping subspaces, even if these subspaces have no theoretical interpretation. Fortunately, Bayes factor model comparison accommodates overlapping models without such difficulties. We argue given that posterior predictive approaches force certain specifications that may not be ideal for scientific questions, they are less desirable in many substantive applications.