Unlinked header file in new PyMC5 install

Questions v5
1

Hello-

I recently tried to move from PyMC3 to PyMC5 on an M1 max (16 inch MacBook pro, OS version 13.2.1) and I seem to be getting a linkage error. I’ve tried this with multiple different models, including several that I copied/pasted directly from the documentation, so I’m pretty sure its not model specific. Any thoughts?

I’ve tried deleting/reinstalling dev tools, deleting/reinstalling my whole conda installation and installing PyMC5 via the linked instructions: Installation — PyMC 5.0.2 documentation

<!-- language: none -->

You can find the C code in this temporary file: /var/folders/wd/wlg9qv65701fzmgdkdfy5vtm0000gr/T/pytensor_compilation_error_29agcezr
Traceback (most recent call last):
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/lazylinker_c.py", line 79, in <module>
    raise ImportError(
ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.212, but found None. Extra debug information: force_compile=False, _need_reload=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/lazylinker_c.py", line 100, in <module>
    raise ImportError(
ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.212, but found None. Extra debug information: force_compile=False, _need_reload=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/burke.227/Documents/research/lecanemab/simpleModel.py", line 61, in <module>
    idata = pm.sample()
            ^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/sampling/mcmc.py", line 452, in sample
    step = assign_step_methods(model, step, methods=pm.STEP_METHODS, step_kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/sampling/mcmc.py", line 194, in assign_step_methods
    return instantiate_steppers(model, steps, selected_steps, step_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/sampling/mcmc.py", line 112, in instantiate_steppers
    step = step_class(vars=vars, model=model, **args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/step_methods/hmc/nuts.py", line 182, in __init__
    super().__init__(vars, **kwargs)
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/step_methods/hmc/base_hmc.py", line 109, in __init__
    super().__init__(vars, blocked=blocked, model=self._model, dtype=dtype, **pytensor_kwargs)
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/step_methods/arraystep.py", line 166, in __init__
    func = model.logp_dlogp_function(vars, dtype=dtype, **pytensor_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/model.py", line 642, in logp_dlogp_function
    ip = self.initial_point(0)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/model.py", line 1126, in initial_point
    fn = make_initial_point_fn(model=self, return_transformed=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/initial_point.py", line 152, in make_initial_point_fn
    func = compile_pymc(inputs=[], outputs=initial_values, mode=pytensor.compile.mode.FAST_COMPILE)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pymc/pytensorf.py", line 1104, in compile_pymc
    pytensor_function = pytensor.function(
                        ^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/compile/function/__init__.py", line 315, in function
    fn = pfunc(
         ^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/compile/function/pfunc.py", line 367, in pfunc
    return orig_function(
           ^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/compile/function/types.py", line 1763, in orig_function
    fn = m.create(defaults)
         ^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/compile/function/types.py", line 1656, in create
    _fn, _i, _o = self.linker.make_thunk(
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/basic.py", line 254, in make_thunk
    return self.make_all(
           ^^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/vm.py", line 1300, in make_all
    vm = self.make_vm(
         ^^^^^^^^^^^^^
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/vm.py", line 1021, in make_vm
    from pytensor.link.c.cvm import CVM
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/cvm.py", line 13, in <module>
    from pytensor.link.c.lazylinker_c import CLazyLinker
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/lazylinker_c.py", line 143, in <module>
    GCC_compiler.compile_str(dirname, code, location=loc, preargs=args)
  File "/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/cmodule.py", line 2641, in compile_str
    raise CompileError(
pytensor.link.c.exceptions.CompileError: Compilation failed (return status=1):
/Users/burke.227/opt/anaconda3/envs/pymc_env/bin/clang++ -dynamiclib -g -Wno-c++11-narrowing -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -undefined dynamic_lookup -I/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/numpy/core/include -I/Users/burke.227/opt/anaconda3/envs/pymc_env/include/python3.11 -I/Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code -L/Users/burke.227/opt/anaconda3/envs/pymc_env/lib -fvisibility=hidden -o /Users/burke.227/.pytensor/compiledir_macOS-13.2.1-x86_64-i386-64bit-i386-3.11.0-64/lazylinker_ext/lazylinker_ext.so /Users/burke.227/.pytensor/compiledir_macOS-13.2.1-x86_64-i386-64bit-i386-3.11.0-64/lazylinker_ext/mod.cpp
In file included from /Users/burke.227/.pytensor/compiledir_macOS-13.2.1-x86_64-i386-64bit-i386-3.11.0-64/lazylinker_ext/mod.cpp:1:
In file included from /Users/burke.227/opt/anaconda3/envs/pymc_env/lib/python3.11/site-packages/pytensor/link/c/c_code/pytensor_mod_helper.h:4:
In file included from /Users/burke.227/opt/anaconda3/envs/pymc_env/include/python3.11/Python.h:23:
/Users/burke.227/opt/anaconda3/envs/pymc_env/bin/../include/c++/v1/stdlib.h:93:15: fatal error: 'stdlib.h' file not found
#include_next <stdlib.h>
              ^~~~~~~~~~
1 error generated.
'''
2

I ran into this problem on my MacBook as well. Here are the steps I followed to resolve it. As I note, I have no idea which step/steps are important. If you approach it more scientifically, please report back.

3

I greatly appreciate the feedback.

However…I’m afraid I haven’t solved the question of which step is most important, I tried all 4 steps and appear to be getting the same error (I’m assuming the .pytensorrc file goes at the root of the .pytensor directory under home) .

Any other suggestions? I’m pretty much an idiot when it comes to C…but, I get the sense that pytensor is mislinked somehow…in spite of my frantic googling, I’m not finding obvious answers…

4

The header error you describe is definitely related to the C-compiler, which is my steps (1) and (3) in the other thread. You’re positive you have the X-Code Command Line Tools installed? This is the most likely source of the problem.

5

Yep, they’re installed.

xcode-select -version xcode-select version 2396.

I even tried deleting them and reinstalling via brew (I did it via Xcode-select --install, previously), thinking there was some chance it might link something slightly differently. But, no dice. Still getting the same error. As I tried to reinstall via brew, I also deleted every file that ‘brew doctor’ mentioned might be an issue…but, no luck. I verified that the other steps are in place…but still, no luck.

6

My (uneducated) guess then is that it’s to do with where the compiler is looking for the standard libraries. If the path fix that worked for me doesn’t work for you, I found other proposed path fixes here and here.

7

Your uneducated guess was a good one. It was a path problem, somehow there was a hard-coded SDKROOT reference in my .zshrc. I didn’t put it there manually…no clue how it got there. But, that fixed it.

Thanks a lot for your help!

1 Like
8

Several people have reported problems getting up and running on M1/2 macs, could you write down exactly what you changed and to what so there’s somewhere to point people to in the future?

9

The specific change that I made was to: 1. delete 2 lines and a comment from my .zshrc file…

# Created by `userpath` on 2022-05-12 22:37:04

#export PATH=“$PATH:/Users/{myusername}/.local/bin”

#export SDKROOT=“/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk”

and, 2. make sure that the SDKROOT pointed to brew:

export LIBRARY_PATH=/opt/homebrew/lib

1 Like