Fast but now slow sample speeds (MacOS)


First, off, I’m excited to use pymc3. It’s been very interesting so far.

I’m using baysesian inference to infer parameters for a simple hydrologic model. Thus I’m using a ‘black-box’ likelihood function that I was able to get working after some trouble-shooting. The ‘forward’ model is written in fortran and imported as a module with f2py (maybe not relevant).

I’ve found some weird behavior with the sample speed, however. My initial tests with two parameters went very quickly, with:

pm.Potential(‘likelihood’, logl(parameters))
with model:
trace = pm.sample()

And everything seemed to be working correctly. And it was really fast, which was cool, and the trace plots of the results looked correct.

But now, as of a few hours later, when I run the same code again, I’m getting very very slow ‘draw’ speeds. It says approximately 1.0 - 1.5 s/draw. I don’t have the number from previously but this is much much slower (like, 100x slower). I did make a few very minor modifications to the forward model, but the runtime and solutions are not significantly different.

I think that my environment must have changed somehow. I’m using conda environments on MacOS 10.15.2. I just reinstalled pymc3 (3.9.3) in a fresh environment but that does not seem to have helped.

Any ideas on what I can check next?

Thanks in advance

Here are two MCMC things that could cause this:

There are a couple sources of randomness when you start running a model by default:

  • Initial point: It’ll start at the mean/median/mode of each parameter, transform that to support on the real line, then add Uniform(-2, 2) noise. I am not sure what the initial point for a pm.Potential is! You might try pm.check_test_point(model) to see where it initializes, and the log probability of that point. You can manually set the start point if you know better.
  • Step size and mass matrix: The algorithm in PyMC3 will automatically pick a step size to get ~85% of proposals accepted, and choose a mass matrix to use for your momentum in the NUTS sampler. We set the mass matrix to be the variance of the posterior which, if we actually knew that, you wouldn’t have to sample. Most of the “tuning” is spent making a reasonable estimate of the variance. This can go wrong in a few ways, one of which is that the variance of most posteriors change depending on where you are (otherwise it is a Gaussian). If you initialize or get stuck in a funny part of the posterior, you might end up adapting to very small momentum, and then need to take tons of leapfrog steps to make any progress.
  • Both these changes might be helped by just running more “chains” (set chains=12 , as in pm.sample(chains=12, discard_tuned_samples=False)). The chains will be initialized and estimate step size/mass matrix independently, so maybe one or two will do a good job. You can then cancel sampling, inspect the trace to find the ones that were actually exploring, and initialize all your chains with those starting points.

Here are more boring things that could cause this

  • Did you change the model at all? The more code you include, the more we may be able to help!
  • Is this a jupyter notebook? I’d check if some variables got overwritten.
  • You reinstalled in a fresh environment, but maybe some libraries changed underneath you. I can’t think of any that should cause a change like that, though.
1 Like

Awesome, thanks for the quick response. As it turns out, the “minor modifications”, (which included two additional parameters) that I made to the forward model were the culprit. I removed those and the speediness returned. So it’s good to know it wasn’t something weird about my environment. Thanks for the other suggestions.