Memory spike at the end of the MCMC sampling

The issue is due to storing pointwise log likelihood values, a step which is done at the end of sampling when calculating ess and rhat. The default is to store such data because it is required for loo/waic calculation and further model comparison.

Taking a look at the answers to

should give more details and guidance on avoiding the issue.

I am also interested in these use cases when due to large number of observations pointwise log likelihood or posterior predictive do not fit in memory. We are working on integrating Dask with ArviZ (see work started on https://github.com/arviz-devs/arviz/pull/1229) to eventually allow ppc checks and loo/waic calculation for these models.

2 Likes