Multiprocessing (windows 10) BrokenPipeError: [Errno 32] Broken pipe

aseyboldt · November 27, 2018, 8:40am

That’s great news! Thanks @lucianopaz!
The reason that windows needs the if __name__ is that on windows multiprocessing creates a completely new process, that then has to import the modules the code is using. On Unix it just forks the process, so it basically copies the original processes memory space (using copy on write so that it can do it fast).
It should work on jupyter, that does stuff internally to make using multiprocessing possible (but if you have a machine that fails that way handy, it would be great if you could try again just to make sure )

So I guess this means that the problem is mostly about bad error messages? Somehow we lose the original (meaningful) RuntimeError or theano error and instead print the BrokenPipeError.
I guess when we receive a message from the other end we should catch BrokenPipeError and try to figure out what happend on the other end. We could also try to make the child process a bit more resilient to errors like that, it shouldn’t die because it can’t compute a gradient.

The relevant code should be here:

github.com

pymc-devs/pymc3/blob/master/pymc3/parallel_sampling.py#L87


def run(self):
    try:
        # We do not create this in __init__, as pickling this
        # would destroy the shared memory.
        self._point = self._make_numpy_refs()
        self._start_loop()
    except KeyboardInterrupt:
        pass
    except BaseException as e:
        e = ExceptionWithTraceback(e, e.__traceback__)
        self._msg_pipe.send(("error", None, e))
    finally:
        self._msg_pipe.close()


def _make_numpy_refs(self):
    shape_dtypes = self._step_method.vars_shape_dtype
    point = {}
    for name, (shape, dtype) in shape_dtypes.items():
        array = self._shared_point[name]
        self._shared_point[name] = array
        point[name] = np.frombuffer(array, dtype).reshape(shape)

If I read the second warning here correctly, then I’m a bit confused about why we don’t get the error message on the main thread. If there is a theano problem, it should send that error message to the main thread and then exit. But why does the main thread just see a Broken Pipe, and not the error massage?

Topic		Replies	Views
@lucianopaz。Please help me solve the Windows multiprocessing issue Questions	33	3232	December 8, 2018
Getting broken pipe error Questions	1	1185	November 3, 2020
Problem with multiprocessing in PyMC3 Questions	5	3722	August 20, 2018
Pymc3 getting stuck after initialization Questions	41	9010	February 9, 2022
Simple example fails with njobs>1 on Windows OS Development bug	6	2253	April 22, 2018

Multiprocessing (windows 10) BrokenPipeError: [Errno 32] Broken pipe

Related topics