SFB 1313 Publication "The Four Ways to Consider Measurement Noise in Bayesian Model Selection—And Which One to Choose"

New SFB 1313 publication (University of Stuttgart), published in Water Resources Research. The paper has been prepared within SFB 1313's research project B04.

"The Four Ways to Consider Measurement Noise in  Bayesian Model Selection - And Which One to Choose"

Authors

• Sebastian Reuschen (University of Stuttgart, SFB 1313 research project B04)
• Wolfgang Nowak (University of Stuttgart, SFB 1313 research projects B04 and B05)
• Anneli Guthke (University of Stuttgart, SimTech)

Abstract

Bayesian model selection (BMS) is a statistically rigorous approach to assess the plausibility of competing models. It naturally accounts for uncertainties in models and data. In this study, we discuss the role of measurement noise in BMS deeper than in past literature. We istinguish between four cases, accounting for noise in models and/or data: (1) no-no, (2) no-yes, (3) yes-no, and (4) yes-yes. These cases differ mathematically and philosophically. Only two out of these four cases are logically consistent, and they represent two potentially conflicting research questions: “Which model is best in modeling the pure physics?” (Case 1) and “which model is best in predicting the data-generating process (i.e., physics plus noise)?” (Case 4). If we are interested in the “pure physics question,” we face two practical challenges: First, we would need noise-free data, which is impossible to obtain; and second, the numerical approximation of Bayesian model evidence can be hard when neglecting noise. We discuss how to address both challenges and reveal that a fallback to the easier “data-generation question” as a proxy for the “physics question” is not appropriate. We demonstrate on synthetic scenarios and a real-world hydrogeological case study that the choice of the case has a significant impact on the outcome of posterior model weights, and hence on results of the model ranking, model selection, model averaging, model confusion analysis, and uncertainty quantification. Reality might force us to use a different case than philosophy would suggest, and we provide guidance on how to interpret model probabilities under such conditions.