I am trying to use black-box likelihood functions with different samplers and I am facing issues with parallelization.
When using DE-Metropolis sampler, I am able to sample multiple chains but only when cores=1 is set, for cores>1 it does not start sampling (the progress bar is stuck). I face the same issue with DE-Metropolis-Z as well as Slice samplers.
Is there a general incompatibility issue with parallelization when a black-box likelihood function is used?
What might be the probable issue/reason for this behavior?
What OS and pymc3 version are you using? Apparently there have been a few issues with parallel processing on Windows recently. See here and make sure you’re using the latest version of pymc3
Hi @Elenchus, I am using Linux Mint 20 “Ulyana” with PyMC3=3.9.3.
Mmmg, I thought Linux was still a safe paradise when it comes to multithreading
Pinging @aseyboldt and @michaelosthege as they know wayyyyy more than me about that
Many multiprocessing issues are related to where the code of your model is written and how it is executed.
- notebooks are generally more problematic - try to move as much code as possible into a separate
. py file that you can import
- don’t execute stuff on 0-level indent. Put everything inside classes and functions
if __name__ == ''__main__"…
- watch the jupyter console for error messages
There can be more problems with external libraries that have multiprocessing issues, but that depends on your model…