Issue with chain parallelization when using black-box likelihood function

Questions
1

I am trying to use black-box likelihood functions with different samplers and I am facing issues with parallelization.

When using DE-Metropolis sampler, I am able to sample multiple chains but only when cores=1 is set, for cores>1 it does not start sampling (the progress bar is stuck). I face the same issue with DE-Metropolis-Z as well as Slice samplers.

Is there a general incompatibility issue with parallelization when a black-box likelihood function is used?
What might be the probable issue/reason for this behavior?

2

What OS and pymc3 version are you using? Apparently there have been a few issues with parallel processing on Windows recently. See here and make sure you’re using the latest version of pymc3

3

Hi @Elenchus, I am using Linux Mint 20 “Ulyana” with PyMC3=3.9.3.

4

Mmmg, I thought Linux was still a safe paradise when it comes to multithreading :thinking:
Pinging @aseyboldt and @michaelosthege as they know wayyyyy more than me about that

5

Many multiprocessing issues are related to where the code of your model is written and how it is executed.

  • notebooks are generally more problematic - try to move as much code as possible into a separate . py file that you can import
  • don’t execute stuff on 0-level indent. Put everything inside classes and functions
  • if __name__ == ''__main__"…
  • watch the jupyter console for error messages

There can be more problems with external libraries that have multiprocessing issues, but that depends on your model…

2 Likes