Saving model and inference results (traces?) in pyMC4

Questions v5
1

Hello pymc community!
I’m running SMC on a cluster computer and would like to save the inference results and model so that I can later access it from my personal computer. I could not find how to do this for pyMC4. There are ways to accomplish this for pyMC3:

  1. using pm.save_trace which does not seem to exist for MC4
  2. using pickle which gives an attribute error (AttributeError: Can’t pickle local object bounded_cont_transform..transform_params’)

I could also not find it in the documentation.What’s the right way to do this?

Thanks for the help!

ps. im using v 4.1.1

2

I will let someone else chime in about saving the model (cloudpickle?). For the sampling results, I would suggest using arviz.InferenceData.to_netcdf(). If you are feeling adventurous, you can try out arviz.InferenceData.to_zarr(), which is a newer format that looks promising (that’s what I have heard, I have no firsthand knowledge!).

1 Like
3

Thanks, this seems to work for saving the inference results. For the model, is it perhaps possible to get out the parameters from the netcdf structure and recreate the last models in the chains (i’m running SMC with 8 chains). Saving with pickle does not seem to work.

4

I think the package has moved to cloudpickle, you might try that.

2 Likes
5

Hey, I just got around to test this. Cloudickle works for saving models and inference results as well, thanks for the input!

I am wondering if it is also possible to save intermediate results during a run. I am running sequential monte carlo with a custom likelihood which takes long to sample, about 10 seconds. My simulations have been running for 3 days and are still going. I am printing out the likelihood value so I can track that the algorithm seems to be doing something right, but I would like to see the results in intermediate steps.

This question was partly asked before here but the answer is not really satisfactory, and the issue not resolved. Any suggestions here? I could also start a new thread for this particular question if it is better. Thanks for the help!

6

I would suggest opening a new topic to ask that. It’s more likely to find the right people that way.