The question is simple. Just curious to know why when I try to install with pip I get the following error:
InvalidVersion: Invalid version: '0.1.43ubuntu1' (package: python-debian)
1 Like
Bringing this back up. Would love a pip installation for latest version.
1 Like
Wait, what? You can install pymc with pip. What problems do you have with the installation?
Back when v4 came out there was lots of discussion on why devs could only get it to work with conda. Official documentation only shows installation steps for conda. I’ve tried it with pip recently and get various errors
My understanding is that installing PyMC via pip is fine. If you do, however, installing all the various pieces of the C backend that PyTensor relies on (for fast sampling) ends up being your responsibility (and installing C compilers and associated tooling is a very platform-/hardware-specific operation). Conda-forge has pre-compiled packages for these components (for various platforms, easing the the creation of functional (i.e., fast) PyMC environments.
2 Likes
I tend to have more luck with pip than with conda these days. With wheels and package caching it’s also so much faster to install. (I never got rid of the BLAS warnings even when working with conda, so for me there is no change in the speed of execution when using pip or conda). The one thing pip cannot do for me is install a different python version. I now have to work on a machine with only python 3.12 installed, and anaconda. The only way I got my pymc code to work was to install python 3.11 with anaconda, and then create a virtual environement from the resulting 3.11 binary. I then forget about conda and work with pip in the virtualenv to install pymc etc (I don’t even have to activate the conda env since the virtualenv already has the correct paths).
Sure, but that can be an incredible speed penalty 
Hi @ricardoV94 , I surely wish I could “incredibly” speed my currently 2-3h long calculation (including 30 min pure compile + NUTS init). Unfortunately since I have not been able to do so even from a brand new conda env as in the install doc, I have nothing to miss! On the contrary, on my most recent attempt on a supercomputer, the pymc with conda install actually stalls entirely on pm.sample, whereas the pip install runs jusst fine ! I checked with numpy.show_config() that my pip install defaults to the system’s “openblas” library:
In [2]: np.show_config()
Build Dependencies:
blas:
detection method: pkgconfig
found: true
include directory: /usr/local/include
lib directory: /usr/local/lib
name: openblas64
openblas configuration: USE_64BITINT=1 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2
pc file directory: /usr/local/lib/pkgconfig
version: 0.3.23.dev
lapack:
detection method: internal
found: true
include directory: unknown
lib directory: unknown
name: dep140213194937296
openblas configuration: unknown
pc file directory: unknown
version: 1.26.4
whereas the stalling conda install indicates:
blas_info:
libraries = ['cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/perrette/.conda/envs/pymc-5.9.0/lib']
include_dirs = ['/home/perrette/.conda/envs/pymc-5.9.0/include']
language = c
define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
libraries = ['cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/perrette/.conda/envs/pymc-5.9.0/lib']
include_dirs = ['/home/perrette/.conda/envs/pymc-5.9.0/include']
language = c
lapack_info:
libraries = ['lapack', 'blas', 'lapack', 'blas']
library_dirs = ['/home/perrette/.conda/envs/pymc-5.9.0/lib']
language = f77
...
And it seems conda installed everything correctly (though possibly redundantly !?):
(.venv-pymc59-py311) [perrette@login04 slr-tidegauges-future]$ ls /home/perrette/.conda/envs/pymc-5.9.0/lib/*lapack* -dl
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/liblapacke.so -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/liblapacke.so.3 -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/liblapack.so -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/liblapack.so.3 -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 27 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libmkl_scalapack_ilp64.so -> libmkl_scalapack_ilp64.so.2
-rwxr-xr-x 5 perrette users 7718768 30 giu 2023 /home/perrette/.conda/envs/pymc-5.9.0/lib/libmkl_scalapack_ilp64.so.2
lrwxrwxrwx 1 perrette users 26 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libmkl_scalapack_lp64.so -> libmkl_scalapack_lp64.so.2
-rwxr-xr-x 5 perrette users 7728456 30 giu 2023 /home/perrette/.conda/envs/pymc-5.9.0/lib/libmkl_scalapack_lp64.so.2
(.venv-pymc59-py311) [perrette@login04 slr-tidegauges-future]$ ls /home/perrette/.conda/envs/pymc-5.9.0/lib/*blas* -dl
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libblas.so -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libblas.so.3 -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libcblas.so -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libcblas.so.3 -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 22 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libopenblas.a -> libopenblasp-r0.3.27.a
-rw-r--r-- 4 perrette users 64261142 8 lug 10.12 /home/perrette/.conda/envs/pymc-5.9.0/lib/libopenblasp-r0.3.27.a
-rwxr-xr-x 4 perrette users 36518408 8 lug 10.12 /home/perrette/.conda/envs/pymc-5.9.0/lib/libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libopenblas.so -> libopenblasp-r0.3.27.so
lrwxrwxrwx 1 perrette users 23 17 set 16.31 /home/perrette/.conda/envs/pymc-5.9.0/lib/libopenblas.so.0 -> libopenblasp-r0.3.27.so
I see the thread WARNING (pytensor.tensor.blas) - #38 by GrantDaly has been updated, I’ll take another look. (the pip-friendly NUMBA backend seems promising).
The JAX backend is also pip friendly. I believe uou can disable the C compiler entirely using pytensor.config flags, then stick to JAX sampling and you’ll have pretty good coverage of PyMC features.
Any reason you’re installing a rather old version of PyMC?
@ricardoV94 Until recently, any higher version would not let me run the code (the code would hang foreever when calling pm.sample). Now I am seeing that python=3.11 + pymc==5.16.2 works on a scaled down version of the model (provided I account for this issue with h5py==3.9.0 and netCDF4==1.6.5), which is encouraging, though I need further testing on the full model.
By the way I just tried NUMBA (and JAX) on the reduced model (10 locations instead of 500+) to no avail. With 5.16.2 I get a few warnings .../pytensor/tensor/rewriting/elemwise.py:1027: UserWarning: Loop fusion failed because the resulting node would exceed the kernel argument limit. and then an error AttributeError: 'Scratchpad' object has no attribute 'ufunc'. I have a vague idea that the warnigns could be linked to calculations in likelihood which involve calculating linear trends (a matrix multiplication) on various locations each with its own start and end year and missing values, and then concatenate these values in one array. In case you can see something useful here, I’ll add this part of the log, FYI:
Apply node that caused the error: Add(Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Composite{...}.2, ExpandDims{axes=[0, 1]}.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Mul.0, Composite{...}.0, ExpandDims{axes=[0, 1]}.0)
Toposort index: 697
Inputs types: [TensorType(float64, shape=(2, None, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, None, None)), TensorType(float64, shape=(2, None, None)), TensorType(float64, shape=(1, 200, None)), TensorType(float64, shape=(1, 1, 10)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, 200, None)), TensorType(float64, shape=(2, None, None)), TensorType(float64, shape=(2, None, None)), TensorType(float64, shape=(1, 200, None)), TensorType(float64, shape=(1, 1, 10))]
Inputs shapes: [(2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (1, 200, 10), (1, 1, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (2, 200, 10), (1, 200, 10), (1, 1, 10)]
Inputs strides: [(16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (160, 160, 16), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (16000, 80, 8), (160, 160, 16)]
Inputs values: ['not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown']
Outputs clients: [[CumOp{1, add}(Add.0), Subtensor{:, start:stop}(Add.0, 93, 120), Subtensor{:, i}(Add.0, 150), Subtensor{:, i}(Add.0, 199), Subtensor{i}(Add.0, 0)]]
You seem to have a rather messy model. That node has 47 inputs. Do you perhaps have an unrolled for loop that should be handled with a scan? Or many many variables that should be batched?
Did you do anything to cxx? Otherwise to avoid that error you need to exclude the fusion rewrite. Somehow when that rewrite aborts we are getting to that AttributeError. If you could provide a reproducible example would be great. I have been hoping to track that error for a while now.
Hi @ricardoV94 , I do have a large model. I thought I’d have batched everything I could (though in at least one instance I do something like pt.concatenate([func(array, i) for i in range(...) ]) when I don’t know better. I’ll see what I can do to produce a reproducible example, but I don’t promise anything because a) it’s time-consuming and b) it does not seem to be the main issue for me. Well I did give it a try to one operation I thought would have caused the misbehaviour but that ran fine with all modes:
import numpy as np
from scipy.linalg import lstsq
import xarray as xa
import pymc as pm
import pytensor
import pytensor.tensor as pt
def pytensor_lintrend(series, mask=None):
# calculate linear trend
n = series.shape[0].eval()
A = np.array([np.arange(n) - n//2, np.ones(n)]).T
if mask is not None:
A = A[~mask]
series = series[~mask]
P = np.linalg.pinv(A.T@A)
x = (P@A.T) @ series
return x[0]
rng = np.random.default_rng(seed=3)
n = 10
l = 118
obs = rng.normal(size=(n, l))
obs_mask = rng.normal(size=(n, l)) > 0
with pm.Model() as model:
rates = pm.Normal('timeseries', shape=(n, l))
cumul = pt.cumsum(rates, axis=-1)
trends = pt.stack([pytensor_lintrend(cumul[i], mask=obs_mask[i]) for i in range(n)])
obs_trends = pt.stack([pytensor_lintrend(pytensor.shared(obs[i]), mask=obs_mask[i]) for i in range(n)]).eval()
pm.Normal('obs', trends, .1, observed=obs_trends)
idata = pm.sample()
However it showed that while the default took 7 seconds to run, both NUMBA and JAX take almost 5 min. Certainly not a good advertisement for these alternative modes on my machine.
PS: the runtime suggests to me that my pip-only install actually uses available hardware optimizations (via openblas). The 2-3 hours runtime is not bad considering I run a pretty large model with 500 locations x time-series (well I only use scalar diagnostic on these time-series) and tens of different variables.
PPS: I checked the newer 5.16.2 runs through but unfortunately not on multiple cores (I experimented with the flags: NUMBA_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 but it did not help).