Simple example fails with njobs>1 on Windows OS

bug

#1

Hi,
I’m trying to run the example on the quickstart tutorial but run into an error, below. If I set the sample.njobs parameter to 1 (i.e., trace = pm.sample(1000, tune=500,njobs=1)) this error does not occur. The temporary file lists a problem with the g++ command:

with pm.Model() as model:
        mu = pm.Normal('mu', mu=0, sd=1)
        obs = pm.Normal('obs', mu=mu, sd=1, observed=np.random.randn(100))
    
        trace = pm.sample(1000, tune=500)
        
        print(len(trace))

#output

C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
WARNING (theano.gof.compilelock): Overriding existing lock by dead process '6068' (I am process '4268')
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
C:\Miniconda3\envs\py35_64\lib\site-packages\pymc3\model.py:384: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if not np.issubdtype(var.dtype, float):
Multiprocess sampling (3 chains in 3 jobs)
NUTS: [mu]
C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

  0%|          | 0/1500 [00:00<?, ?it/s]
You can find the C code in this temporary file: C:\Users\daniel\AppData\Local\Temp\theano_compilation_error_q1z142e3

forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source             
libifcoremd.dll    00007FFB32AA94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFB62687EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFB63791FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFB660BEFC1  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source             
libifcoremd.dll    00007FFB32AA94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFB62687EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFB63791FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFB660BEFC1  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source             
libifcoremd.dll    00007FFB32AA94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFB62687EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFB63791FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFB660BEFC1  Unknown               Unknown  Unknown
forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source             
libifcoremd.dll    00007FFB32AA94C4  Unknown               Unknown  Unknown
KERNELBASE.dll     00007FFB62687EDD  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFB63791FE4  Unknown               Unknown  Unknown
ntdll.dll          00007FFB660BEFC1  Unknown               Unknown  Unknown

Problem occurred during compilation with the command line below:
"C:\Miniconda3\envs\py35_64\Library\mingw-w64\bin\g++.exe" -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -march=broadwell -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=9216 -mtune=generic -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -DMS_WIN64 -I"C:\Miniconda3\envs\py35_64\lib\site-packages\numpy\core\include" -I"C:\Miniconda3\envs\py35_64\include" -I"C:\Miniconda3\envs\py35_64\lib\site-packages\theano\gof\c_code" -L"C:\Miniconda3\envs\py35_64\libs" -L"C:\Miniconda3\envs\py35_64" -o "C:\Users\daniel\AppData\Local\Theano\compiledir_Windows-10-10.0.16299-SP0-Intel64_Family_6_Model_158_Stepping_10_GenuineIntel-3.5.5-64\tmprxnfwonr\mbf603e414396181d78ebdc7b7f3292b9a15629083c2e0aa4d31d71596ce9c1a4.pyd" "C:\Users\daniel\AppData\Local\Theano\compiledir_Windows-10-10.0.16299-SP0-Intel64_Family_6_Model_158_Stepping_10_GenuineIntel-3.5.5-64\tmprxnfwonr\mod.cpp" -lpython35

Any ideas?

Thanks


#2

Concerning the error regarding the compilation error, that folder (tmprxnfwonr) is not present on the machine, though there are many others. I assume this is why the compilation fails. Why is that folder missing?

Thanks


#3

This is pretty weird - I haven’t seen it before. What is your configuration? Also theano version?


#4

Yes, what’s even more weird is that the above same code (njobs=3, by default) does, in fact, sometimes work:

C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
WARNING (theano.gof.compilelock): Overriding existing lock by dead process '12572' (I am process '10316')
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (3 chains in 3 jobs)
NUTS: [mu]
C:\Miniconda3\envs\py35_64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

  0%|          | 0/1500 [00:00<?, ?it/s]
  0%|          | 1/1500 [00:00<09:26,  2.64it/s]
 19%|#9        | 290/1500 [00:00<00:01, 605.42it/s]
 42%|####2     | 631/1500 [00:00<00:00, 1088.59it/s]
 66%|######6   | 992/1500 [00:00<00:00, 1459.24it/s]
 89%|########9 | 1340/1500 [00:00<00:00, 1717.84it/s]
100%|##########| 1500/1500 [00:00<00:00, 1817.85it/s]
1000

But on a rerun it usually fails like in the first post. I’m running Miniconda with Python35 (I’ve previously tried with Miniconda/Python36 with the same issue) on Windows 10; below is a list of all packages installed:

# packages in environment at C:\Miniconda3\envs\py35_64:
#
backports                 1.0                      py35_1    conda-forge
backports.functools_lru_cache 1.5                      py35_0    conda-forge
ca-certificates           2018.1.18                     0    conda-forge
certifi                   2018.1.18                py35_0    conda-forge
cycler                    0.10.0                   py35_0    conda-forge
freetype                  2.8.1                    vc14_0  [vc14]  conda-forge
h5py                      2.7.1                    py35_2    conda-forge
hdf5                      1.10.1                   vc14_2  [vc14]  conda-forge
icc_rt                    2017.0.4             h97af966_0  
icu                       58.2                     vc14_0  [vc14]  conda-forge
intel-openmp              2018.0.0             hd92c6cd_8  
joblib                    0.11                     py35_0    conda-forge
joblib                    0.11                      <pip>
jpeg                      9b                       vc14_2  [vc14]  conda-forge
libgpuarray               0.7.5                    vc14_0  [vc14]  conda-forge
libpng                    1.6.34                   vc14_0  [vc14]  conda-forge
libpython                 2.1                      py35_0  
m2w64-binutils            2.25.1                        5  
m2w64-bzip2               1.0.6                         6  
m2w64-crt-git             5.0.0.4636.2595836               2  
m2w64-gcc                 5.3.0                         6  
m2w64-gcc-ada             5.3.0                         6  
m2w64-gcc-fortran         5.3.0                         6  
m2w64-gcc-libgfortran     5.3.0                         6  
m2w64-gcc-libs            5.3.0                         7  
m2w64-gcc-libs-core       5.3.0                         7  
m2w64-gcc-objc            5.3.0                         6  
m2w64-gmp                 6.1.0                         2  
m2w64-headers-git         5.0.0.4636.c0ad18a               2  
m2w64-isl                 0.16.1                        2  
m2w64-libiconv            1.14                          6  
m2w64-libmangle-git       5.0.0.4509.2e5a9a2               2  
m2w64-libwinpthread-git   5.0.0.4634.697f757               2  
m2w64-make                4.1.2351.a80a8b8               2  
m2w64-mpc                 1.0.3                         3  
m2w64-mpfr                3.1.4                         4  
m2w64-pkg-config          0.29.1                        2  
m2w64-toolchain           5.3.0                         7  
m2w64-tools-git           5.0.0.4592.90b8472               2  
m2w64-windows-default-manifest 6.4                           3  
m2w64-winpthreads-git     5.0.0.4634.697f757               2  
m2w64-zlib                1.2.8                        10  
mako                      1.0.7                    py35_0    conda-forge
markupsafe                1.0                      py35_0    conda-forge
matplotlib                2.1.2                    py35_0    conda-forge
mkl                       2018.0.1             h2108138_4  
mkl-service               1.1.2            py35h051acba_4  
msys2-conda-epoch         20160418                      1  
numpy                     1.14.1           py35h4a99626_1  
openssl                   1.0.2n                   vc14_0  [vc14]  conda-forge
pandas                    0.22.0                   py35_0    conda-forge
patsy                     0.5.0                    py35_0    conda-forge
pip                       9.0.1                    py35_1    conda-forge
pygpu                     0.7.5                    py35_0    conda-forge
pymc3                     3.3                       <pip>
pyparsing                 2.2.0                    py35_0    conda-forge
pyqt                      5.6.0                    py35_4    conda-forge
python                    3.5.5                         0    conda-forge
python-dateutil           2.6.1                    py35_0    conda-forge
pytz                      2018.3                     py_0    conda-forge
qt                        5.6.2                    vc14_1  [vc14]  conda-forge
scipy                     1.0.0            py35h75710e8_0  
seaborn                   0.8.1                    py35_0    conda-forge
setuptools                38.5.1                   py35_0    conda-forge
sip                       4.18                     py35_1    conda-forge
six                       1.11.0                   py35_1    conda-forge
statsmodels               0.8.0                    py35_0    conda-forge
Theano                    1.0.1+44.g7b648eb94           <pip>
tornado                   4.5.3                    py35_0    conda-forge
tqdm                      4.19.6                     py_0    conda-forge
vc                        14                            0    conda-forge
vs2015_runtime            14.0.25420                    0    conda-forge
wheel                     0.30.0                   py35_2    conda-forge
wincertstore              0.2                      py35_0    conda-forge
zlib                      1.2.11                   vc14_0  [vc14]  conda-forge

Any idea about the fortran error (forrtl: error (200): program aborting due to control-C event)? I’ve google posts about it being related to scipy but that doesn’t seem to be the case here.

Thanks


#5

It seems to be an issue on Windows (with the processes specifically). The same code on my Miniconda3 install on Linux works fine.


#6

The same for me, when the code run in Pycharm under Win10, njobs must be 1. But sometimes it works when njobs >1 in Jupyter notebook under Win10.


#7

Try deleting your local Theano directory before you start your kernel and run your model. Make sure it’s only the local theano directory and not the main one. Worked for me.

C:\Users\username\AppData\Local\Theano directory

This theano folder will actually recreate itself every time you open a new jupyter notebook and run pymc3. So the solution is only a bandade and not long-term.