I am thinking about setting a some hyperparameter tuning code for selecting PyMC models. This is not for prediction or causal inference, but rather for prior predictive selection and estimator validation. I have a set of evaluation objectives for (1) matching background knowledge, (2) recovering the correct parameters, and (3) sampling nicely. I want to search for models that are Pareto efficient for these evaluation criteria.
MCMC sampling tends to go slowly when the model is poorly specified. Naturally, a blind optimizer searching through a wide variety of models is bound to try some badly-specified models. Only spending a fixed amount of time on each model sounds like a way to curtail this problem.
It appears that PyMC’s pymc.sample has a callback parameter. I’m wondering how I would write such a callback to stop sampling and return the trace after a certain amount of time has elapsed.
Would such a callback be the best way to achieve running models for a fix maximum time? If so, how would I write such a callback? If not, is there anything else I should try?