How to verify that uncertainty (estimated from pymc3) is accurate?

Questions
9

Thanks a lot for your suggestions, @gbernstein! My plan is to validate pymc3 on simulated data before using it on the real data.

Thanks for the concise summary below. It is very helpful for me to understand the evaluation procedure.
image

I have a few follow up questions:

  1. How can I draw the (theta_i, x_i)? I understand the language but how do I put it in python to generate (theta_i, x_i) given a pre-defined prior p(theta) and likelihood p(x|theta)?
  2. What is U_i? Again given F_i(theta) how can I calculate Ui?
  3. What is a good choice for M?

Thanks a lot and really appreciate your feedback!