Multiprocessing problems on linux

Questions v5
1

I’m on a fresh (as of last night) reinstall of Ubuntu 24.04.4 LTS, and pyMC (v 5.27.1) is giving me multiprocessing errors. I followed the recommended installation via Anaconda (conda 26.1.0), with python 3.14.3. It’s definitely not a problem with my code, as even running the introductory tutorial produces the error.

    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Setting cores to 1 fixes the problem, and everything else seems to work fine.

2

Maybe I’m over-indexing on the mention of “fork” in the error, but it seems like the wrong multiprocessing context is being used. Can you try doing pm.sample(..., mp_ctx='forkserver') ?

3

I get the same error sampling with mp_ctx='forkserver'.

4

We run the test suite on ubuntu python 3.14 so I’m at a bit of a loss. Tagging @aseyboldt since he’s the multiprocessing guru. There’s also mp_ctx='spawn' as the other option to try.

You should also be able to sidestep this with a non-python NUTS sampler: pm.sample(nuts_sampler='nutpie') for example.

5

Using nutpie works. Should I mark that as solved? It’s sufficient for my purposes, but the underlying problem still exists.

6

I’d leave it open since it’s not actually resolved. But I suspect the answer is something like “multiprocessing in python sucks”. I really hope the nogil build sorts this all out in the next ~12 months.

7

@jessegrabowski I’m curious what nutpie does differently that solves the problem. When you call pm.sample(nuts_sampler=’nutpie’) does nutpie take a single pickleable python object (all the needed info and parameters needed to sample) and send it to rust where rayon handles the parallel chains?

8

@aseyboldt is the right person to ask for the technical details. My superficial understanding is that it compiles a numba function, then gives rust a pointer to where that function lives in memory so it can be called. The actual sampler nutpie uses is called nuts-rs, and indeed uses rayon.

1 Like