Problem with multiprocessing in PyMC3

Hi all,

first of all thanks for the PyMC3 package, I’ll use it for implementation of Kennedy O’Hagan implementation and it seems like just the right tool.

However, I do have some troubles with current version of PyMC3. As I suspect it is a bug of PyMC3 but rather a wrong usage on my side, I hope this is the right plattform to post this issue.

  • PyMC3 Version: 3.4.1-py36_0
  • Theano Version: 1.0.2-py36he980bc4_0
  • Python Version: 3.6.6
  • Operating system: Windows 10 64bit (also tested Ubuntu 18.04 64bit, if traceback needed I’ll reproduce it, getting also an error)
  • How did you install PyMC3: conda (full package list of new environment, see below)

I’m trying to run your “simpletest.py” example. Running it with no further adjustmens gives the first traceback. adding cores=1 to the pm.sample method (pm.samle(n, cores=1)) runs smoothly (2nd traceback).

So my (probably valid) assumption is that it is caused by multiprocessing, but what am I doing wrong or missing here? Any help very much appreciated :slight_smile:

Traceback (sorry for german language) is:

(pymc_env) Notes\stuff>python simpletest.py
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [z_logodds__, x]

You can find the C code in this temporary file: C:\Users\pre\AppData\Local\Temp\theano_compilation_error_i91anzzq
Traceback (most recent call last):
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\joblib\externals\loky\process_executor.py", line 377, in
_process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\multiprocessing\queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\compile\function_module.py", line 1082, in _constr
uctor_Function
    f = maker.create(input_storage, trustme=True)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\compile\function_module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\vm.py", line 1091, in make_all
    impl=impl))
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\op.py", line 955, in make_thunk
    no_recycling)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cc.py", line 1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cmodule.py", line 1181, in module_from_key
    module = lnk.compile_cmodule(location)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cc.py", line 1523, in compile_cmodule
    preargs=preargs)
  File "D:\Programme\anaconda3\envs\pymc_env\lib\site-packages\theano\gof\cmodule.py", line 2391, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('The following error happened while compiling the node', Alloc(TensorConstant{(1,) of 0.0}, Shape_i{0}.0), '
\n', 'Compilation failed (return status=3): ', '[Alloc(TensorConstant{(1,) of 0.0}, <TensorType(int64, scalar)>)]')
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
libifcoremd.dll    00007FFC620E94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFCCA6C7EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFCCD4D1FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFCCD60F061  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
libifcoremd.dll    00007FFC620E94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFCCA6C7EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFCCD4D1FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFCCD60F061  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
libifcoremd.dll    00007FFC620E94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFCCA6C7EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFCCD4D1FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFCCD60F061  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
libifcoremd.dll    00007FFC620E94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFCCA6C7EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFCCD4D1FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFCCD60F061  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source
libifcoremd.dll    00007FFC620E94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFCCA6C7EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFCCD4D1FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFCCD60F061  Unknown               Unknown  Unknown
FEHLER: Der Prozess "13288" wurde nicht gefunden.

Running it only on single core

(pymc_env) \stuff>python simpletest.py
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Sequential sampling (2 chains in 1 job)
NUTS: [z_logodds__, x]
100%|████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:01<00:00, 1069.47it/s]
100%|████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:00<00:00, 1783.11it/s]

Conda output:

conda create -n pymc_env python=3.6 pymc3

Package Plan

environment location: D:\Programme\anaconda3\envs\pymc_env

added / updated specs:
- pymc3
- python=3.6

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
python-3.6.6               |       hea74fb7_0        21.6 MB

The following NEW packages will be INSTALLED:

blas:                           1.0-mkl
ca-certificates:                2018.03.07-0
certifi:                        2018.4.16-py36_0
cycler:                         0.10.0-py36h009560c_0
freetype:                       2.8-h51f8f2c_1
h5py:                           2.8.0-py36h3bdd7fb_0
hdf5:                           1.10.2-hac2f561_1
icc_rt:                         2017.0.4-h97af966_0
icu:                            58.2-ha66f8fd_1
intel-openmp:                   2018.0.3-0
joblib:                         0.12-py36_0
jpeg:                           9b-hb83a4c4_2
kiwisolver:                     1.0.1-py36h12c3424_0
libgpuarray:                    0.7.6-hfa6e2cd_0
libpng:                         1.6.34-h79bbb47_0
libpython:                      2.1-py36_0
m2w64-binutils:                 2.25.1-5
m2w64-bzip2:                    1.0.6-6
m2w64-crt-git:                  5.0.0.4636.2595836-2
m2w64-gcc:                      5.3.0-6
m2w64-gcc-ada:                  5.3.0-6
m2w64-gcc-fortran:              5.3.0-6
m2w64-gcc-libgfortran:          5.3.0-6
m2w64-gcc-libs:                 5.3.0-7
m2w64-gcc-libs-core:            5.3.0-7
m2w64-gcc-objc:                 5.3.0-6
m2w64-gmp:                      6.1.0-2
m2w64-headers-git:              5.0.0.4636.c0ad18a-2
m2w64-isl:                      0.16.1-2
m2w64-libiconv:                 1.14-6
m2w64-libmangle-git:            5.0.0.4509.2e5a9a2-2
m2w64-libwinpthread-git:        5.0.0.4634.697f757-2
m2w64-make:                     4.1.2351.a80a8b8-2
m2w64-mpc:                      1.0.3-3
m2w64-mpfr:                     3.1.4-4
m2w64-pkg-config:               0.29.1-2
m2w64-toolchain:                5.3.0-7
m2w64-tools-git:                5.0.0.4592.90b8472-2
m2w64-windows-default-manifest: 6.4-3
m2w64-winpthreads-git:          5.0.0.4634.697f757-2
m2w64-zlib:                     1.2.8-10
mako:                           1.0.7-py36he15cdb7_0
markupsafe:                     1.0-py36h0e26971_1
matplotlib:                     2.2.2-py36h153e9ff_1
mkl:                            2018.0.3-1
mkl-service:                    1.1.2-py36h57e144c_4
mkl_fft:                        1.0.1-py36h452e1ab_0
mkl_random:                     1.0.1-py36h9258bd6_0
msys2-conda-epoch:              20160418-1
numpy:                          1.14.5-py36h9fa60d3_0
numpy-base:                     1.14.5-py36h5c71026_0
openssl:                        1.0.2o-h8ea7d77_0
pandas:                         0.23.1-py36h830ac7b_0
patsy:                          0.5.0-py36_0
pip:                            10.0.1-py36_0
pygpu:                          0.7.6-py36h452e1ab_0
pymc3:                          3.4.1-py36_0
pyparsing:                      2.2.0-py36h785a196_1
pyqt:                           5.9.2-py36h1aa27d4_0
python:                         3.6.6-hea74fb7_0
python-dateutil:                2.7.3-py36_0
pytz:                           2018.4-py36_0
qt:                             5.9.6-vc14h62aca36_0
scipy:                          1.1.0-py36h672f292_0
setuptools:                     39.2.0-py36_0
sip:                            4.19.8-py36h6538335_0
six:                            1.11.0-py36h4db2310_1
sqlite:                         3.24.0-h7602738_0
theano:                         1.0.2-py36he980bc4_0
tornado:                        5.0.2-py36_0
tqdm:                           4.23.4-py36_0
vc:                             14-h0510ff6_3
vs2015_runtime:                 14.0.25123-3
wheel:                          0.31.1-py36_0
wincertstore:                   0.2-py36h7fe50ca_0
zlib:                           1.2.11-h8395fce_2

The older implementation use Joblib for multiprocessing, which sometimes creates pickling issue. This problem is recently addressed in PR #3011 thanks to @aseyboldt

Could you please try installing PyMC3 from master and run the code again? We will also do a release shortly.

1 Like

Hey @junpenglao @aseyboldt it worked! :slight_smile: thank you!!!

1 Like

Hi, I have exactly the same problem on my new Windows pc (strangely, it works on my old one). Getting pretty much the same error for, coincidentally, Kennedy O’Hagan code. I tried to install pymc3 from master via

pip install git+https://github.com/pymc-devs/pymc3@master

But it did not make it work…

Got a similar issue. May this issue be related?

You are right, that seems to be very much the same problem.