Hi,
Facing issue while running pm.sample
“Multiprocess sampling (2 chains in 2 jobs)
NUTS: [gamma, alpha, beta_comp, beta, decay, peak, noise_var, tau]
Traceback (most recent call last):
File “C:\Users\deepakkapoor\Desktop\mmx_streamlit\app.py”, line 60, in
media_model = MediaModelBis(ctrl_model, [‘online_sales’])
File “C:\Users\deepakkapoor\Desktop\mmx_streamlit\media_model\media.py”, line 23, in init
self.fit_result, self.fit_summary, self.trace, self.loo, self.waic = fit_model(self.mmm_model, return_all = True)
File “C:\Users\deepakkapoor\Desktop\mmx_streamlit\media_model\sampling.py”, line 12, in fit_model
trace = pm.sample(int(n_samples/chains), tune=int(n_tune_samples/chains), chains=chains,
File “C:\Users\deepakkapoor\anaconda\envs\pymc3_env5\lib\site-packages\deprecat\classic.py”, line 215, in wrapper_function
return wrapped_(*args_, **kwargs_)
File “C:\Users\deepakkapoor\anaconda\envs\pymc3_env5\lib\site-packages\pymc3\sampling.py”, line 575, in sample
trace = _mp_sample(**sample_args, **parallel_args)
File “C:\Users\deepakkapoor\anaconda\envs\pymc3_env5\lib\site-packages\pymc3\sampling.py”, line 1480, in _mp_sample
sampler = ps.ParallelSampler(
File “C:\Users\deepakkapoor\anaconda\envs\pymc3_env5\lib\site-packages\pymc3\parallel_sampling.py”, line 423, in init
step_method_pickled = pickle.dumps(step_method, protocol=-1)
TypeError: cannot pickle ‘fortran’ object”
Things I have tried so far →
- Setting cores = 1
- Updating joblib
- Installing pymc3 using this link : Installation Guide (Linux) · pymc-devs/pymc Wiki · GitHub
I get the same error. This is on OS X. The following code:
#!/usr/bin/env python
import bambi, pandas
df = pandas.DataFrame({"y": [1, 3], "group": ["a", "b"]})
model = bambi.Model("y ~ group", df)
model.fit(cores=4)
Results in this traceback:
WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [y_sigma, Intercept, group]
Traceback (most recent call last):
File "error.py", line 7, in <module>
model.fit(cores=4)
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/bambi/models.py", line 325, in fit
return self.backend.run(
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/bambi/backend/pymc.py", line 96, in run
result = self._run_mcmc(
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/bambi/backend/pymc.py", line 172, in _run_mcmc
idata = pm.sample(
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/pymc/sampling/mcmc.py", line 766, in sample
_mp_sample(**sample_args, **parallel_args)
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/pymc/sampling/mcmc.py", line 1141, in _mp_sample
sampler = ps.ParallelSampler(
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/pymc/sampling/parallel.py", line 400, in __init__
step_method_pickled = cloudpickle.dumps(step_method, protocol=-1)
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/Users/terry/miniconda3/envs/310/lib/python3.10/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle 'fortran' object
This is on OS X 13.4.1 with Python 3.10 and pymc 5.6.1.
Thanks for any help!
Terry
1 Like
Welcome!
Does it work when you try model.fit(cores=1)? How did you install PyMC? Did you follow the installation instructions found here?
Hi. Thanks for the reply. Yes, the code worked fine with cores=1. I had PyMC installed via pip.
I’ve somehow managed to get myself into a state where the clang in my environment cannot find unistd.h (sometimes stdio.h), though I have no idea how. When I run the installation instructions you point to, making a brand new env and then running this simple pymc code pymctest.py · GitHub (taken from GLM: Linear regression — PyMC 5.6.1 documentation) I get errors such as:
pytensor.link.c.exceptions.CompileError: Compilation failed (return status=1):
/Users/terry/miniconda3/envs/pymc_env/bin/clang++ -dynamiclib -g -Wno-c++11-narrowing -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -undefined dynamic_lookup -I/Users/terry/miniconda3/envs/pymc_env/lib/python3.11/site-packages/numpy/core/include -I/Users/terry/miniconda3/envs/pymc_env/include/python3.11 -I/Users/terry/miniconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code -L/Users/terry/miniconda3/envs/pymc_env/lib -fvisibility=hidden -o /Users/terry/.pytensor/compiledir_macOS-13.4.1-x86_64-i386-64bit-i386-3.11.4-64/lazylinker_ext/lazylinker_ext.so /Users/terry/.pytensor/compiledir_macOS-13.4.1-x86_64-i386-64bit-i386-3.11.4-64/lazylinker_ext/mod.cpp
In file included from /Users/terry/.pytensor/compiledir_macOS-13.4.1-x86_64-i386-64bit-i386-3.11.4-64/lazylinker_ext/mod.cpp:1:
In file included from /Users/terry/miniconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code/pytensor_mod_helper.h:4:
In file included from /Users/terry/miniconda3/envs/pymc_env/include/python3.11/Python.h:23:
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/stdlib.h:150:34: error: unknown type name 'ldiv_t'
inline _LIBCPP_INLINE_VISIBILITY ldiv_t div(long __x, long __y) _NOEXCEPT {
^
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/stdlib.h:151:12: error: no member named 'ldiv' in the global namespace
return ::ldiv(__x, __y);
~~^
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/stdlib.h:154:34: error: unknown type name 'lldiv_t'
inline _LIBCPP_INLINE_VISIBILITY lldiv_t div(long long __x,
^
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/stdlib.h:156:12: error: no member named 'lldiv' in the global namespace
return ::lldiv(__x, __y);
~~^
In file included from /Users/terry/.pytensor/compiledir_macOS-13.4.1-x86_64-i386-64bit-i386-3.11.4-64/lazylinker_ext/mod.cpp:1:
In file included from /Users/terry/miniconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code/pytensor_mod_helper.h:4:
In file included from /Users/terry/miniconda3/envs/pymc_env/include/python3.11/Python.h:26:
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/string.h:95:102: error: unknown type name 'size_t'
inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_PREFERRED_OVERLOAD const void* memchr(const void* __s, int __c, size_t __n) {
^
/Users/terry/miniconda3/envs/pymc_env/bin/../include/c++/v1/string.h:98:90: error: unknown type name 'size_t'
inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_PREFERRED_OVERLOAD void* memchr(void* __s, int __c, size_t __n) {
^
In file included from /Users/terry/.pytensor/compiledir_macOS-13.4.1-x86_64-i386-64bit-i386-3.11.4-64/lazylinker_ext/mod.cpp:1:
In file included from /Users/terry/miniconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code/pytensor_mod_helper.h:4:
/Users/terry/miniconda3/envs/pymc_env/include/python3.11/Python.h:29:12: fatal error: 'unistd.h' file not found
# include <unistd.h>
^~~~~~~~~~
7 errors generated.
Sorry for the long output. This happens whether I use Python 3.10 or 3.11 and whether I use conda or mamba.
So I’m a bit stuck. When I use brew to install clang, its version of clang has no problem compiling a tiny C program that #includes unistd.h. But the clang in my conda environment fails to find the header. I’ve been looking (for many hours now) online to see if I can find a solution, without success (there are of course many such errors for various packages going back many years).
Thanks again!
I got a new (M2) macbook and did a system migration from my old machine. I hoped the problem would be gone, but it wasn’t.
I have since installed Xcode and did the following horrible hack, which works. After installing pymc in my conda environment, I have two versions of clang. The one in the environment and the Xcode version in /usr/bin. I wondered whether the Xcode version might be fine, so I went into the bin dir in the conda environment and replaced clang and clang++ with symbolic links to the versions in /usr/bin. I’m not going to put the commands here, because I don’t want anyone else to just paste my commands into their shell without knowing what they’re doing. If you do know what you’re doing, you will know how to do what I just described 
You can also leave me a message here.
Summary: using the xcode version of clang works fine. It’s also a horrible hack and is likely to cause an issue at some later point and need to be undone. For now though I can at least use pymc again. I have no idea why the conda installed version of clang cannot find the standard headers. All web pages I read about this give solutions for Ubuntu, not OS X.
I thought that the fortran object stuff had been mostly resolved with better support for blas libraries on Mac via conda-forge installations. The fortran object thing had already popped up some time ago, and I had gotten in touch with people over at scipy/numpy to see if this was something that they thought they should handle on their end or not. The responses were very clear, fortran objects wont be pickleable because fortran objects can store very weird stuff, not only function pointers.
The root cause of the problem is that at some point in pytensor we are trying to grab a fortran object and serialize it in order to send it to another process that should run it. At the time I first tried to deal with this, I had no idea how to track down what was going on nor how to truly fix it. Some weeks ago, I had to poke around the codebase again to try to understand blas linkage better and I think that I might have found what is causing the fortran objects to get “pickled”. In these lines, pytensor stores the raw fortran objects in a dictionary that is keyed by the tensor dtype. Later on, the Gemv Op tries to access the dictionary in its perform method. Since we are using cloudpickle, the object’s method is pickled by value and not by reference. I think that this means that cloudpickle looks at the objects that are actually referenced by the perform method, and tries to pickle them as well in order to get a serialized version of the method that would run identically to the original one regardless of where the method got called (i.e. another process on another machine, using a different OS, etc). And at that point, pickling fails because fortran objects cannot be pickled. I think that the solution will be to use a higher level interface to actually get the underlying fortran function pointers instead of actually storing their values in a dictionary. I’ll try to open a draft PR to do this, using scipy.linalg.blas.get_blas_funcs.
From my latest poking around with pytensor while trying to get a reproducible code that fails, I think that I found the culprit of the issue you’ve been running into. It isn’t the Gemv Op as I wrote before, it looks like it’s Ger_scipy. For some reason that Op actually stores the Fortran object as an attribute to call it later during perform. That makes the whole Op impossible to pickle and raises this criptic error. I’ll work on a fix to push up to pytensor. Once that happens, @terrycojones, we might ask you to rerun your code using a newer version of pytensor.
BTW the failure may be a symptom of a bad installation. I think we only fallback to the scipy implementation when we don’t have a proper linking to BLAS directly?
That may be why @lucianopaz couldn’t reproduce the problem locally
@terrycojones, this PR should fix the underlying issue. But as @ricardoV94 said, having this problem usually means that for some reason, pytensor does not know where to find the BLAS libraries to link with directly. I don’t have a Mac so I can’t properly check this stuff, but usually conda installations bring BLAS stuff that you can use directly.
Could I ask you if you have the vecLib libraries in your M2 Mac? If you do, what do you get if you try to run clang --print-search-dirs? I would like to understand where the frameworks are installed by default so that I can get pytensor to find the accelerate framework libraries automatically.