How to verify that uncertainty (estimated from pymc3) is accurate?

If you’re working with fully synthetic data for which you can simulate and run many trials then Cook 2006 introduces a method to evaluate the calibration of a posterior, i.e. whether the posterior properly quantifies uncertainty.

See the Evaluation paragraph in Section 4 here for a succinct summary.

1 Like