Nested Parallel in PyMC3

Questions
1

Dear PyMC community,

My forward model (written by Fortran) is quite complex and time-consuming to calculate. Therefore, I have to use parallel in my forward model. However, these conflicts with PyMC3 parallel sampling. I am wondering if there is a way to allow the multiprocessing in PyMC3 sampling while my forward model can still be done through parallel computation.

Also, do you have any advice on speeding up PyMC3 for the time-consuming forward model? Many thanks for your help.

Best regards,

Hongwei

2

Have you try sampling with core=1? This will disable the pymc3 parallel sampling so it does not conflict your custom forward computation (or at least surface the bug if there is any)

3

Hi Junpeng,

Thanks for the reply. In the case where I use core =1, it works well. But the entire process becomes very slow since there are many cores that are not used during sampling. I am wondering if there is a way to sample using multicores while my forward model is also using multiprocessing.

The multiprocessing in my forward model was implemented using Python Ray library. When I set pm.sample with core number larger than 2, the sampling stuck at the initial stage and not using any cores.

Thanks for your time again.

Hongwei

4

Maybe try controlling the number of cores forward model use? I am not sure how well it will works. We sometimes observe similar multiprocess race issue using GP, where some linear algebra operation is multi-threaded

5

You can take a look of the following to see if there is something useful:

6

Hi Junpeng,

Thank you very much for your help. I will see if a different implementation of parallelization can solve my problem.

Thanks again,

Hongwei