I’m trying to develop PyMC on some pretty outdated MacBook (MacOS 12.7 and no option to upgrade to newer versions) and run into some problems more protracted than was described in Pytensor compilation error. Related to that (because I’m trying to hack PyTensor’s c compilation machinery to fix this issue), but also more generally, I’m trying to change code in PyTensor, then activate PyMC’s conda environment, then run
pip install --no-build-isolation --no-deps -e .
from the PyTensor’s directory (this idea is sourced from this comment).
However, this doesn’t seem to work: after executing the pip install above python -c "import sys; print('\n'.join(sys.path))" gives
And pytensor is already present in /usr/local/Caskroom/miniconda/base/envs/pymc-test-jax/lib/python3.11/site-packages and therefore the “local” installation doesn’t take effect.
What is the “standard” of developing PyTensor in conjunction with PyMC? I couldn’t find the workflow documented anywhere or discussed in this forum.
Secondly, I want to raise an issue of how incredibly difficult it is to make all this stuff work, as evidenced in Pytensor compilation error and also in my experiments. I’d like to invite people to brainstorm ideas for reducing this burden to make contributions to PyMC easier. For example, maybe drop the C backend, and rely solely on JAX (and PyTorch later, as per Help wanted: Continue PyTorch backend for PyTensor)? Would it alleviate the setup burden, or would not change things much? Are there some cases in which C will be faster than JAX? Are there concerns that JAX only supports 64-bit platforms and PyMC wants to maintain 32-bit platform support (if it actually still does)?
JAX is not supported on windows, which cuts out a large number of users. It also has major limitations: lack of support for dynamic array size, and limited support for linking to legacy parts of the scientific stack (LAPACK, for instance). Basically it’s not very “hackable”. We also have a numba backend which is quite nice, and doesn’t suffer from these issues (although it does have it’s own).
For development, I recommend that you fork and clone pytensor. I just use the CI .yaml files in the github actions directory to set up the development environment. Then you can pip install your local pytensor directory if you want to use that in a PyMC project.
It’s true that installation was quite difficult in the past, but this has largely been resolved as far as I can tell. Compiler errors like the one linked are usually indicative of installation errors (which usually boil down to “you didn’t use use conda but should have”, but sometimes also involve x-code stuff on Mac). If you’re hitting them, please share your installation procedure so we can fix it.
By forking PyTensor, do you mean changing files in my checked-out pytensor directory (such as name in pyproject.toml, and maybe some other places?), then installing it as as a whole separate package my-pytensor, and then swapping the dependency in pymc’s code?
Yes, change files in your local pytensor project, then in your PyMC environment pip install the local pytensor directory.
We’re not planning to fully replace the C backend anytime soon, and if we did, JAX is not 100% aligned with our needs, at least for the reasons already stated above.