I noticed in that thread, jack stated
" the graph compilation succeeds either by disabling multiprocessing or by setting theano.config.mode = 'FAST_COMPILE'
What is the risk to the model if I do that and where do I do that? In the model code?
You should try it out to see if that works. The modes in theano more or less leverage if your model will take a longer time to be compiled, because it will try to optimize the underlying binary to be able to run fast, or if it just compiles fast without many optimizations. The latter maybe will cause the model to run slower (but it may be negligible to what you gain with running chains in parallel).
Where do I actually type that in though?
The documentation says that you can specify the mode, which is part of theano.config, in one of three ways:
The order of precedence is:
- an assignment to theano.config.<property>
- an assignment in
THEANO_FLAGS- an assignment in the .theanorc file (or the file indicated in
THEANORC)
To do the first method, in your script (or notebook) you have to do:
... # The imports
import theano
theano.config.mode = 'FAST_COMPILE'
... # The rest of your model and sampling code
The second method requires you to set environment variables on windows.
The third method involves writing custom options in the .theanorc hidden file.
theano.config.mode = 'FAST_COMPILE' is usually much slower than the default, use with care.
@jordan.howell2, I just made a branch that does not move compiled theano.function's around. Could you check it out and see if you still get the error? I can’t reproduce the error you get on my machine, so I cannot test if it fixes the problem or not directly.
1 Like
Can you verify that I’m suppose to type:
pip install git+https://github.com/lucianopaz/pymc3/tree/differ_step_compile/pymc3
I recommend you create a new conda environment, install theano first and then use the pip install like this
pip install git+https://github.com/lucianopaz/pymc3.git@differ_step_compile
1 Like
This is the new error:
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Sampling 4 chains: 0%| | 0/22000 [00:00<?, ?draws/s]
You can find the C code in this temporary file: C:\Users\jorda\AppData\Local\Temp\theano_compilation_error_bnjab9wy
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFE55963B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFEA10D56FD Unknown Unknown Unknown
KERNEL32.DLL 00007FFEA20C3034 Unknown Unknown Unknown
ntdll.dll 00007FFEA4A33691 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFE55963B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFEA10D56FD Unknown Unknown Unknown
KERNEL32.DLL 00007FFEA20C3034 Unknown Unknown Unknown
ntdll.dll 00007FFEA4A33691 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFE55963B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFEA10D56FD Unknown Unknown Unknown
KERNEL32.DLL 00007FFEA20C3034 Unknown Unknown Unknown
ntdll.dll 00007FFEA4A33691 Unknown Unknown Unknownpymc3.parallel_sampling.RemoteTraceback:
“”"
Traceback (most recent call last):
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\parallel_sampling.py”, line 110, in run
self._start_loop()
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\parallel_sampling.py”, line 160, in _start_loop
point, stats = self._compute_point()
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\parallel_sampling.py”, line 191, in _compute_point
point, stats = self._step_method.step(self._point)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\step_methods\arraystep.py”, line 270, in step
self._init_fs()
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\step_methods\hmc\base_hmc.py”, line 97, in _init_fs
super()._init_fs()
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\step_methods\arraystep.py”, line 251, in _init_fs
self.vars, dtype=self._dtype, **self._theano_kwargs)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\model.py”, line 721, in logp_dlogp_function
return ValueGradFunction(self.logpt, grad_vars, extra_vars, **kwargs)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\model.py”, line 728, in logpt
logp = tt.sum([tt.sum(factor) for factor in factors])
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\model.py”, line 728, in
logp = tt.sum([tt.sum(factor) for factor in factors])
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\tensor\basic.py”, line 3210, in sum
out = elemwise.Sum(axis=axis, dtype=dtype, acc_dtype=acc_dtype)(input)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 670, in call
no_recycling=)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 955, in make_thunk
no_recycling)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 858, in make_c_thunk
output_storage=node_output_storage)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1217, in make_thunk
keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1157, in compile
keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1624, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cmodule.py”, line 1152, in module_from_key
src_code = lnk.get_src_code()
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1500, in get_src_code
mod = self.get_dynamic_module()
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1544, in get_dynamic_module
self.code_gen()
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 840, in code_gen
behavior = op.c_code(node, name, isyms, osyms, sub)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\tensor\elemwise.py”, line 1559, in c_code
code = “\n”.join(self._c_all(node, name, inames, onames, sub))
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\tensor\elemwise.py”, line 1415, in _c_all
var = Elemwise(scalar.identity)(input)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 670, in call
no_recycling=)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 955, in make_thunk
no_recycling)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\op.py”, line 858, in make_c_thunk
output_storage=node_output_storage)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1217, in make_thunk
keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1157, in compile
keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1624, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cmodule.py”, line 1189, in module_from_key
module = lnk.compile_cmodule(location)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cc.py”, line 1527, in compile_cmodule
preargs=preargs)
File “C:\Users\jorda\Anaconda3\lib\site-packages\theano\gof\cmodule.py”, line 2396, in compile_str
(status, compile_stderr.replace(‘\n’, '. ')))
Exception: ('Compilation failed (return status=1): C:\Users\jorda\AppData\Local\Theano\compiledir_Windows-10-10.0.17134-SP0-Intel64_Family_6_Model_158_Stepping_10_GenuineIntel-3.7.1-64\tmptq8h9tcz\mod.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in\r. #include <Python.h>\r. ^\r. ', ‘[Elemwise{identity}(__logp_mu_a)]’)
“”"The above exception was the direct cause of the following exception:
Exception: ('Compilation failed (return status=1): C:\Users\jorda\AppData\Local\Theano\compiledir_Windows-10-10.0.17134-SP0-Intel64_Family_6_Model_158_Stepping_10_GenuineIntel-3.7.1-64\tmptq8h9tcz\mod.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in\r. #include <Python.h>\r. ^\r. ', ‘[Elemwise{identity}(__logp_mu_a)]’)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “D:\Sales\sales_bayesian2.py”, line 78, in
trace = pm.sample(draws=5000, init=‘advi’, progressbar=True)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\sampling.py”, line 438, in sample
trace = _mp_sample(**sample_args)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\sampling.py”, line 980, in _mp_sample
for draw in sampler:
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\parallel_sampling.py”, line 393, in iter
draw = ProcessAdapter.recv_draw(self._active)
File “C:\Users\jorda\Anaconda3\lib\site-packages\pymc3\parallel_sampling.py”, line 297, in recv_draw
raise error from old_error
RuntimeError: Chain 1 failed.
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFE55963B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFEA10D56FD Unknown Unknown Unknown
KERNEL32.DLL 00007FFEA20C3034 Unknown Unknown Unknown
ntdll.dll 00007FFEA4A33691 Unknown Unknown Unknown
Well, so at least now the processes spawn and manage to communicate their exception to the parent process, which is a plus (although completely unsatisfactory!).
Theano still cannot compile the model’s logp and Jacobian functions for the same 64bit reason. The thing is that you can compile them in the main process when you sample with just a single core! I don’t understand why the spawn has problems. I’ll try to ask the theano guys
@jordan.howell2, no, sorry. Thank theano is no longer being actively maintained and they haven’t had time look into the issue.
I’m very confused about your particular problem. You are able to compile and run 64 bit models with cores=1, but when you make a process pool, they can no longer compile the models. To me, this indicated that the spawned processes were not using the same compile options or compile than the main process. I thought this was due to a change in the environment variables or conda environment but my tests on windows did not show any undesired change of the kind. Maybe it’s because I’m just not able to reproduce the error you observe. But in the end it’s very hard to blind debug. I’ll try keep searching a but next week
Will it help if I attach the jupyter notebook and data?
Yes, it will help, but I still think the problem is related to your setup environment.
All the info I could find about 64 bit complaint says that it is caused because the 32 bit compiler was installed instead of the 64 bit or things like that: 1, 2, 3, 4 and 5.
If you upload your notebook and data I’ll take a look
Here you go @lucianopaz.sales_bayesian.py (3.2 KB)
Feature Files can be found here:
https://candidtruthdsccom-my.sharepoint.com/:x:/g/personal/jordan_howell_candidtruthdsc_com/EZhuimxLaulOjnTuSqF5Ii0B20WzO5GadgCmbkV69bs6nA?e=Qcs219
Store File can be found here:
https://candidtruthdsccom-my.sharepoint.com/:x:/g/personal/jordan_howell_candidtruthdsc_com/EW66foSpX7hGt6wWo1OUUwoBSLO_PbCh0ZJO9QQ2wIZy4A?e=FP1yQD
Train File can be found here:
https://candidtruthdsccom-my.sharepoint.com/:x:/g/personal/jordan_howell_candidtruthdsc_com/EVc7fBM2vlpFoKgmibr56LEBittZ8teuIUvMuHUM3Zuskg?e=daWOgC
The only thing you’ll have to change in the script is where you pull the data from once you download.
Thanks again.
-update. Setting theano to fast compile also yielded an error.
Thanks for the code and data @jordan.howell2. However, I still can’t reproduce your error 
. Your code runs fine (after adding an if __name__ == "__main__") across multiple cores using the pymc3’s master branch on my machine. I think that you must be running into an installation issue or maybe theano finding another c compiler (and python headers) instead of the one you installed through conda.
I would try uninstalling theano, pymc3, pygpu and running:
conda install numpy scipy mkl-service libpython m2w64-toolchain
conda install theano pygpu
conda install pymc3
As a side note, my only change to your script was adding an if __name__ == "__main__": before sampling:
with pm.Model() as sales_model:
# define the priors
mu_a = pm.Normal('mu_a', mu=0, sd=10)
sigma_a = pm.HalfCauchy('sigma_a', 5)
# mu_d = pm.Normal('mu_d', mu = 0, sd = 10)
# sigma_d = pm.HalfCauchy('sigma_d', 5)
alpha = pm.Normal('intercept', mu=mu_a, sd=sigma_a, shape=n_dept)
beta_2 = pm.Normal('IsHoliday_T', mu=0, sd=20)
beta_3 = pm.Normal('Week', mu=0, sd=20)
s = pm.HalfCauchy('sd', 5)
# define the likelihood
mu = alpha[dept_idx] + beta_2 * train['IsHoliday_True'].values + beta_3 * train['Week'].values
y = pm.StudentT('sales', nu=5, mu=mu, sd=s
, observed=train['Weekly_Sales'], shape=train['Weekly_Sales'].shape)
if __name__ == "__main__":
with sales_model:
trace = pm.sample(draws=5000, init='advi', progressbar=True)
print(sales_model.check_test_point())
trace_plot = pm.traceplot(trace[1000:], grid=True)
ppc = pm.plot_posterior(trace[1000:])
summary = pm.summary(trace)
posterior = pm.trace_to_dataframe(trace)
Hi @lucianopaz. This did not work. I finally gave up, partitioned off some space and installed ubuntu. Now it’s running faster than when it did work on Windows. I may have to rethink this whole windows thing.
Thank you so much stick with me. I appreciate the great help.
2 Likes
I’m sorry that I couldn’t help fix the root problem! This windows multiprocessing issue affects many more users. In the end, if we still cannot sort it out, eventually when pymc4 gets released this shouldn’t be a problem anymore.