A Probabilistic Framework for Robust Cross-Validation under Model Misspecification

Thesis Abstract

The reliability of cross-validation as a model evaluation tool deteriorates when candidate models are misspecified, leading to optimistic or pessimistic estimates of predictive performance and ultimately misguided inferences in applied statistics and data science. This study addresses the problem by developing a probabilistic framework for robust cross-validation that explicitly accounts for model misspecification and parameter uncertainty. The aim is to formulate, validate, and demonstrate a principled approach that yields stable predictive assessments across a spectrum of plausible models, thereby enhancing generalizability in empirical settings. Specific objectives are (i) to formalize a probabilistic error-aggregation mechanism that integrates over misspecification uncertainty using a hierarchical prior structure; (ii) to derive theoretical guarantees for the proposed cross-validation criteria, including bias reduction and variance control under misspecification; (iii) to implement an algorithmic pipeline that operationalizes robust cross-validation (RCV) across linear, generalized linear, and nonparametric regression frameworks; (iv) to evaluate RCV performance on synthetic data with controlled misspecification and on real-world data from environmental and health domains; and (v) to compare RCV against standard K-fold and nested cross-validation in terms of predictive accuracy, calibration, and model selection consistency. The methodological approach adopts a mixed-methods design anchored in probabilistic modeling, simulation experiments, and empirical validation. The population comprises statistical models used for predictive tasks across domains where misspecification is common, including linear regression with omitted nonlinearities, logistic regression with mis-specified link functions, and Gaussian process surrogates under non-stationarity. A series of synthetic datasets (n = 500, 1000, 2000 observations) are generated to induce controlled misspecification scenarios, varying the degree of nonlinearity, interaction effects, and noise structure. Two real-world datasets are used a meteorological dataset with n ? 15,000 observations for rainfall forecasting and a health-surveillance dataset with n ? 25,000 records for disease incidence modeling. Data collection instruments are archival in nature (publicly available meteorological records and health surveillance repositories). The RCV framework is constructed within a Bayesian hierarchical model, where misspecification is treated as latent random effects with priors informed by domain knowledge and diagnostic checks. The analysis employs regression techniques (linear, logistic, and generalized additive models), Bayesian model averaging, and cross-validated predictive likelihood as primary evaluation metrics. The method of analysis includes posterior predictive checks, debiased risk estimation, and bootstrap-calibrated uncertainty quantification to assess robustness of cross-validation estimates. The theoretical core leverages established results from misspecified models and Bayesian model evidence, integrating them into a novel probabilistic criterion for cross-validation that remains coherent under misspecification. Key expected findings indicate that the proposed RCV framework reduces over-optimism common to conventional cross-validation when misspecification is present, yielding more stable estimated predictive performance across model classes. In simulation, RCV is anticipated to exhibit lower mean squared predictive error and more accurate calibration curves compared with K-fold and nested cross-validation, particularly under high degrees of misspecification. In empirical applications, RCV is expected to select models with superior out-of-sample predictive accuracy and better-calibrated probability forecasts, while providing interpretable uncertainty about model risk due to misspecification. The study contributes to knowledge by bridging probabilistic modeling and cross-validation practice, offering a generalizable framework that can be adapted to diverse modeling contexts and settings where model misspecification cannot be neglected. It also advances methodological rigor in model assessment by providing a principled way to quantify and propagate misspecification uncertainty through cross-validation procedures, thereby improving decision-making in predictive analytics. The main conclusion is that incorporating misspecification-aware uncertainty into cross-validation improves the reliability and interpretability of predictive performance assessments, especially in complex, real-world data environments. Recommendations include adopting robust cross-validation as a standard diagnostic component in applied modeling pipelines, extending the framework to accommodate non-stationarity and streaming data, and developing software implementations with scalable Bayesian updating to facilitate practical adoption by practitioners and researchers.