Dear PyMC community, My forward model (written by Fortran) is quite complex and time-consuming to calculate. Therefore, I have to use parallel in my forward model. However, these conflicts with PyMC3 parallel sampling. I am wondering if there is a way to allow the multiprocessing in PyMC3 sampling …

Have you try sampling with core=1? This will disable the pymc3 parallel sampling so it does not conflict your custom forward computation (or at least surface the bug if there is any)

Hi Junpeng, Thanks for the reply. In the case where I use core =1, it works well. But the entire process becomes very slow since there are many cores that are not used during sampling. I am wondering if there is a way to sample using multicores while my forward model is also using multiprocessing. …

Maybe try controlling the number of cores forward model use? I am not sure how well it will works. We sometimes observe similar multiprocess race issue using GP, where some linear algebra operation is multi-threaded

You can take a look of the following to see if there is something useful: [image] PYMC3 Multiprocessing issue on a kubernetes cloud Questions The source of the other threads will not be multiprocessing I think, but either openmp through theano if you have big datasets some…

Hi Junpeng, Thank you very much for your help. I will see if a different implementation of parallelization can solve my problem. Thanks again, Hongwei

Nested Parallel in PyMC3

Questions

junpenglao December 6, 2020, 8:38am 5

You can take a look of the following to see if there is something useful:

Topic		Replies	Views
Pm.sample gets stuck after init with cores > 1 Questions	17	4154	January 4, 2021
Does anyone know of any guides or docs for parallel sampling in current version of PYMC (mp_ctx) v5	3	1307	July 24, 2023
Cores not optimally used version agnostic bug	16	279	November 26, 2024
Sampling doesn't start when njobs > 1 for some models Questions	20	4898	January 17, 2020
Sample with multiple cores Questions	3	1547	September 10, 2020

Nested Parallel in PyMC3

Related topics