Firstly, thanks a lot for developing and contribute to PyMC3!
Straight to the point: I have been wondering if it would be possible to run PyMC3 in HPC environments, like super-computers with several nodes. Anyone with experiences to share? It would be of great help to me.
The thing is: I have a really hard and computationally intensive problem. I solve differential equations with a lot of parameters, all fancy stuff like that, etc, etc. Imagine now how long each realization takes… well, so far SMC has been saving me and my group since it runs nicely in parallel (in a single node machine). But I did it successfully only in a single machine with a lot of cores. However, we have a large super-computer available for us, and initial tests have failed.
I did some investigation and found that, behind SMC sampling,
multiprocessing is being used. This is a pretty nice lib, but it can’t handle multi-nodes machine AFAIK. However, there is hope: ray and its multiprocessing implementation. With a single line change in the code (more precisely, in the
multiprocessing can be used in a cluster by providing a proper configuration file (which is a user responsibility). What do you PyMC3-devs think about it? Would it be worth?
I have been thinking about trying to do some contributions to PyMC3. In case you like the above idea, it would be a pleasure to open a PR from my side