Pymc 5.16.2 Hangs on Import

Questions v5
1

Hi everyone, i’ve run into an issue with getting pymc to work on my computer. I am using a Macbook Air, Apple M2 Chip. Fresh computer, which has never had pymc installed on it before.

Similar to this post I have recently installed pymc in a fresh conda environment using the installation guidelines. But when I try to import the package either via a jupyter notebook or via Visual Studio Code the script hangs. To troubleshoot I have tried restarting my computer, reinstalling the environment, this answer here. Of which I have had little success.

This is my current environment, note that when I install using conda create -c conda-forge -n pymc_env "pymc>=5" I get python 3.12.5. Below is the output of conda list

# Name                    Version                   Build  Channel
appnope                   0.1.4              pyhd8ed1ab_0    conda-forge
arviz                     0.19.0             pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
atk-1.0                   2.38.0               h4bec284_2    conda-forge
blas                      2.122                  openblas    conda-forge
blas-devel                3.9.0           22_osx64_openblas    conda-forge
brotli                    1.1.0                h00291cd_2    conda-forge
brotli-bin                1.1.0                h00291cd_2    conda-forge
bzip2                     1.0.8                hfdf4475_7    conda-forge
c-ares                    1.33.1               h44e7173_0    conda-forge
ca-certificates           2024.8.30            h8857fd0_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.5.0              pyhd8ed1ab_0    conda-forge
cairo                     1.18.0               h37bd5c4_3    conda-forge
cctools_osx-64            986                  h0282933_3    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
clang                     17.0.6          default_he371ed4_6    conda-forge
clang-17                  17.0.6          default_hb173f14_6    conda-forge
clang_impl_osx-64         17.0.6              h1af8efd_19    conda-forge
clang_osx-64              17.0.6              hb91bd55_19    conda-forge
clangxx                   17.0.6          default_he371ed4_6    conda-forge
clangxx_impl_osx-64       17.0.6              hc3430b7_19    conda-forge
clangxx_osx-64            17.0.6              hb91bd55_19    conda-forge
cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
compiler-rt               17.0.6               h1020d70_2    conda-forge
compiler-rt_osx-64        17.0.6               hf2b8a54_2    conda-forge
cons                      0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.3.0           py312hc5c4d5f_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
debugpy                   1.8.5           py312h5861a67_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
dm-tree                   0.1.8           py312hf6c9040_4    conda-forge
etuples                   0.3.9              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
executing                 2.1.0              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h73e2aa4_0    conda-forge
filelock                  3.15.4             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h5bb23bf_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.53.1          py312hbd25219_0    conda-forge
freetype                  2.12.1               h60636b9_2    conda-forge
fribidi                   1.0.10               hbcb3906_0    conda-forge
gdk-pixbuf                2.42.12              ha587570_0    conda-forge
graphite2                 1.3.13            h73e2aa4_1003    conda-forge
graphviz                  12.0.0               he14ced1_0    conda-forge
gtk2                      2.24.33              h2c15c3c_5    conda-forge
gts                       0.7.6                h53e17e3_4    conda-forge
h5netcdf                  1.3.0              pyhd8ed1ab_0    conda-forge
h5py                      3.11.0          nompi_py312hfc94b03_102    conda-forge
harfbuzz                  9.0.0                h098a298_1    conda-forge
hdf5                      1.14.3          nompi_h687a608_105    conda-forge
icu                       75.1                 h120a0e1_0    conda-forge
importlib-metadata        8.4.0              pyha770c72_0    conda-forge
importlib_metadata        8.4.0                hd8ed1ab_0    conda-forge
ipykernel                 6.29.5             pyh57ce528_0    conda-forge
ipython                   8.27.0             pyh707e725_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.2              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py312hb401068_0    conda-forge
kiwisolver                1.4.5           py312hc5c4d5f_2    conda-forge
krb5                      1.21.3               h37d8d59_0    conda-forge
lcms2                     2.16                 ha2f27b4_0    conda-forge
ld64_osx-64               711                  hf158dd0_3    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libabseil                 20240116.2      cxx17_hf036a51_1    conda-forge
libaec                    1.1.3                h73e2aa4_0    conda-forge
libblas                   3.9.0           22_osx64_openblas    conda-forge
libbrotlicommon           1.1.0                h00291cd_2    conda-forge
libbrotlidec              1.1.0                h00291cd_2    conda-forge
libbrotlienc              1.1.0                h00291cd_2    conda-forge
libcblas                  3.9.0           22_osx64_openblas    conda-forge
libclang-cpp17            17.0.6          default_hb173f14_6    conda-forge
libcurl                   8.9.1                hfcf2730_0    conda-forge
libcxx                    18.1.8               hd876a4e_6    conda-forge
libcxx-devel              18.1.8               h7c275be_6    conda-forge
libdeflate                1.21                 hfdf4475_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 h10d778d_2    conda-forge
libexpat                  2.6.2                h73e2aa4_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgd                     2.3.3               h2e77e4f_10    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_3    conda-forge
libgfortran5              13.2.0               h2873a65_3    conda-forge
libglib                   2.80.3               h736d271_2    conda-forge
libhwloc                  2.11.1          default_h456cccd_1000    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
libintl                   0.22.5               hdfe23c8_3    conda-forge
libjpeg-turbo             3.0.0                h0dc2134_1    conda-forge
liblapack                 3.9.0           22_osx64_openblas    conda-forge
liblapacke                3.9.0           22_osx64_openblas    conda-forge
libllvm17                 17.0.6               hbedff68_1    conda-forge
libnghttp2                1.58.0               h64cf6d3_1    conda-forge
libopenblas               0.3.27          openmp_h8869122_1    conda-forge
libpng                    1.6.43               h92b6c6a_0    conda-forge
librsvg                   2.58.3               h902c40a_0    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.46.0               h1b8f9f3_0    conda-forge
libssh2                   1.11.0               hd019ec5_0    conda-forge
libtiff                   4.6.0                h603087a_4    conda-forge
libwebp-base              1.4.0                h10d778d_0    conda-forge
libxcb                    1.16                 h00291cd_1    conda-forge
libxml2                   2.12.7               heaf3512_4    conda-forge
libzlib                   1.3.1                h87427d6_1    conda-forge
llvm-openmp               18.1.8               h15ab845_1    conda-forge
llvm-tools                17.0.6               hbedff68_1    conda-forge
logical-unification       0.4.6              pyhd8ed1ab_0    conda-forge
macosx_deployment_target_osx-64 10.13                hbc8f3bb_1    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
matplotlib-base           3.9.2           py312h0d5aeb7_0    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
minikanren                1.0.3              pyhd8ed1ab_0    conda-forge
mkl                       2023.2.0         h54c2260_50500    conda-forge
mkl-service               2.4.0           py312h2402a68_1    conda-forge
multipledispatch          0.6.0              pyhd8ed1ab_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.5                  hf036a51_1    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
numpy                     1.26.4          py312he3a82b2_0    conda-forge
openblas                  0.3.27          openmp_hba01982_1    conda-forge
openjpeg                  2.5.2                h7310d3a_0    conda-forge
openssl                   3.3.1                hd23fc13_3    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py312h1171441_1    conda-forge
pango                     1.54.0               h115fe74_2    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
pcre2                     10.44                h7634a1b_2    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.4.0          py312hbd70edc_0    conda-forge
pip                       24.2               pyh8b19718_1    conda-forge
pixman                    0.43.4               h73e2aa4_0    conda-forge
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
psutil                    6.0.0           py312hbd25219_0    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.3              pyhd8ed1ab_0    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pymc                      5.16.2               hd8ed1ab_0    conda-forge
pymc-base                 5.16.2             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.4              pyhd8ed1ab_0    conda-forge
pytensor                  2.25.4          py312h9592d4c_0    conda-forge
pytensor-base             2.25.4          py312h81f2e74_0    conda-forge
python                    3.12.5          h37a9e06_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-graphviz           0.20.3             pyh717bed2_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyzmq                     26.2.0          py312h54d5c6a_0    conda-forge
qhull                     2020.2               h3c5361c_5    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
scipy                     1.14.1          py312he82a568_0    conda-forge
setuptools                72.2.0             pyhd8ed1ab_0    conda-forge
sigtool                   0.1.3                h88f4db0_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tapi                      1100.0.11            h9ce4665_0    conda-forge
tbb                       2021.12.0            h37c8870_4    conda-forge
threadpoolctl             3.5.0              pyhc1e730c_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
tornado                   6.4.1           py312hb553811_1    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h8827d51_1    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
xarray                    2024.7.0           pyhd8ed1ab_0    conda-forge
xarray-einstats           0.7.0              pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               h0dc2134_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zeromq                    4.3.5                hde137ed_4    conda-forge
zipp                      3.20.1             pyhd8ed1ab_0    conda-forge
zlib                      1.3.1                h87427d6_1    conda-forge
zstd                      1.5.6                h915ae27_0    conda-forge

Does anyone have any ideas as to why this might be happening? Or a specific installation of pymc that its known to work on my computer set-up. Any help would be greatly appreciated :slight_smile: .

Cheers,
Andrew

2

Does it also hang when you import from terminal (with the environment activated)?

3

Interestingly, it imports successfully (and quickly) in the mac os terminal but hangs when I do it in the python VSC terminal. Below are the steps I used in both terminals.

I know it works on my mac os terminal as when I try the command print(f"PyMC version: {pm.__version__}") I get the correct output: PyMC version: 5.16.2

Reproducible steps:
conda activate pymc_env
python3
import pymc as pm
print(f"PyMC version: {pm.__version__}") #Will only execute if it doesn't hang on importing pymc

When I manually terminate in VSC I get the following traceback

Traceback (most recent call last):
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/configparser.py", line 303, in fetch_val_for_key
    return self._pytensor_cfg.get(section, option)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/configparser.py", line 759, in get
    d = self._unify_values(section, vars)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/configparser.py", line 1132, in _unify_values
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'blas'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/configparser.py", line 418, in __get__
    val_str = cls.fetch_val_for_key(self.name, delete_key=delete_key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/configparser.py", line 307, in fetch_val_for_key
    raise KeyError(key)
KeyError: 'blas__ldflags'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pymc/__init__.py", line 46, in <module>
    __set_compiler_flags()
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pymc/__init__.py", line 29, in __set_compiler_flags
    import pytensor
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/__init__.py", line 119, in <module>
    from pytensor import scalar, tensor
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/tensor/__init__.py", line 103, in <module>
    import pytensor.tensor.rewriting
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/tensor/rewriting/__init__.py", line 2, in <module>
    import pytensor.tensor.rewriting.blas
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/tensor/rewriting/blas.py", line 85, in <module>
    from pytensor.tensor.blas import (
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/tensor/blas.py", line 103, in <module>
    from pytensor.tensor.blas_headers import blas_header_text, blas_header_version
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/tensor/blas_headers.py", line 1006, in <module>
    if not config.blas__ldflags:
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/configparser.py", line 422, in __get__
    val_str = self.default()
              ^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2875, in default_blas_ldflags
    return check_libs(
           ^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2794, in check_libs
    res = try_blas_flag(flags)
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2009, in try_blas_flag
    res = GCC_compiler.try_compile_tmp(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2407, in try_compile_tmp
    return cls._try_compile_tmp(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 1885, in _try_compile_tmp
    args = cls.compile_args()
           ^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2291, in compile_args
    default_compilation_result, default_execution_result = try_march_flag(
                                                           ^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2044, in try_march_flag
    compilation_result, execution_result = GCC_compiler.try_compile_tmp(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 2407, in try_compile_tmp
    return cls._try_compile_tmp(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/link/c/cmodule.py", line 1911, in _try_compile_tmp
    out, err, p_ret = output_subprocess_Popen([exe_path])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/site-packages/pytensor/utils.py", line 200, in output_subprocess_Popen
    out = p.communicate()
          ^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/subprocess.py", line 2115, in _communicate
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc/lib/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Thanks for getting back to me quickly. I’m not sure why there is this difference in Visual Studio Code vs the In-built mac terminal

4

Perhaps @maresb or @lucianopaz can give some input on that traceback

5

The traceback that you pasted looks incomplete. I suppose that it should have ended with a KeyboardInterrupt and you might have left that part out. From what I do see, it looks like vscode hanged while it tried to spawn a subprocess to compile a test script using the clang compiler. This might be a problem related to how vscode interacts with subprocess, but I honestly haven’t found any informative errors related to that. The closest I’ve got were this and this.
Having said that, I don’t really know what’s making your vscode hang. It might be that vscode isn’t grabbing the proper conda env where pymc and the clang compiler is installed, or it might be some strange behavior that vscode has to interact with the OS and it might not be playing nicely with python’s subprocess.
If your code works well while running from the mac terminal or from a jupyterlab tab in the browser (or old fashioned jupyter notebook) then the most likely cause is some strange vscode behavior.

1 Like
6

Thanks, Lucianopaz and RicardoV94 for your help. I was hoping it wasn’t a vscode issue but that seems to confirm it.

Based on your last paragraph I went through and completely deleted vscode. Now pymc seems to import fine :slight_smile:. I really appreciate both of your help during this.

I have tested it and have successfully been able to finish one the the pymc GLM module. It seems i get a broken pipe error when I try and idata = sample(3000) without setting cores=1 but this seems to be a common issue (here, here). Ideally, I would like multiprocessing to work but I think at this point I’ll just take a working package to be a win!

I appreciate all the help, if I cannot find a way to get multiprocessing to work on this MacBook would the team suggest I create another question here on the discourse forum?

Thanks,
Andrew

7

Could you share the script that caused the broken pipe and the error you saw? In general, broken pipes come from scripts that try to use multiprocessing without guarding for if __name__ == “__main__” or for trying to use stuff that’s only defined in main and not visible to slanted processes. Usually, you just need small changes to the script to fix everything.
If it still fails, you can install both numba and nutpie and use nutpie as the nuts sampler.

1 Like
8

Sure. Actually, in my original script the guard clause wasn’t working but I think I found the issue, I was calling an arviz plot for the trace which seemed to be the issue. I’ll show the issue ’

the following is a minimally reproducible example:

import numpy as np
import pandas as pd
import arviz as az

import pymc as pm

print('*** Start script ***')
print(f'{pm.__name__}: v. {pm.__version__}')

if __name__ == '__main__':
    SEED = 20180730
    np.random.seed(SEED)

    # Generate data
    mu_real = 0
    sd_real = 1
    n_samples = 1000
    y = np.random.normal(loc=mu_real, scale=sd_real, size=n_samples)

    # Bayesian modelling
    with pm.Model() as model:
    
        mu = pm.Normal('mu', mu=0, sigma=10)
        sd = pm.HalfNormal('sd', sigma=10)
    
        # Likelihood
        likelihood = pm.Normal('likelihood', mu=mu, sigma=sd, observed=y)    
        trace = pm.sample(chains=8, cores=8, random_seed=SEED)

    print('Done!')

This works fine and does infact run as intended giving the following terminal messages (each time the text is printed an extra time is because during the multiprocessing process the script gets recalled so any statements are printed x times where x is the number of jobs?)

(pymc_env) uqamcka3@x86_64-apple-darwin13 Random % /opt/miniconda3/envs/pymc_env/bin/python /Users/uqamcka3/PHD/Projects/Random/multitest.py
*** Start script ***
pymc: v. 5.16.2
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (8 chains in 8 jobs)
NUTS: [mu, sd]
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
*** Start script ***
pymc: v. 5.16.2
Sampling 8 chains, 0 divergences ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 / 0:00:01
Sampling 8 chains for 1_000 tune and 1_000 draw iterations (8_000 + 8_000 draws total) took 91 seconds.
Done!

However when I add a call to arviz to plot a trace by adding az.plot_trace(trace, figsize=(10,7)) at the end of the final line in the reproducible example (outside of the if loop) I get a broken pipe error similar to the issues which were linked - here is the example and traceback:

import numpy as np
import pandas as pd
import arviz as az

import pymc as pm

print('*** Start script ***')
print(f'{pm.__name__}: v. {pm.__version__}')

if __name__ == '__main__':
    SEED = 20180730
    np.random.seed(SEED)

    # Generate data
    mu_real = 0
    sd_real = 1
    n_samples = 1000
    y = np.random.normal(loc=mu_real, scale=sd_real, size=n_samples)

    # Bayesian modelling
    with pm.Model() as model:
    
        mu = pm.Normal('mu', mu=0, sigma=10)
        sd = pm.HalfNormal('sd', sigma=10)
    
        # Likelihood
        likelihood = pm.Normal('likelihood', mu=mu, sigma=sd, observed=y)    
        trace = pm.sample(chains=8, cores=8, random_seed=SEED)

    print('Done!')

az.plot_trace(trace, figsize=(10,7)) # <-- added 

(pymc_env) uqamcka3@x86_64-apple-darwin13 Random % /opt/miniconda3/envs/pymc_env/bin/python /Users/uqamcka3/PHD/Projects/Random/multitest.py
*** Start script ***
pymc: v. 5.16.2
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (8 chains in 8 jobs)
NUTS: [mu, sd]
*** Start script ***
pymc: v. 5.16.2
Traceback (most recent call last):
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/forkserver.py", line 274, in main
    code = _serve_one(child_r, fds,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/forkserver.py", line 313, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/uqamcka3/PHD/Projects/Random/multitest.py", line 32, in <module>
    az.plot_trace(trace, figsize=(10,7))
                  ^^^^^
NameError: name 'trace' is not defined. Did you forget to import 'trace'?
Traceback (most recent call last):
  File "/Users/uqamcka3/PHD/Projects/Random/multitest.py", line 28, in <module>
    trace = pm.sample(chains=8, cores=8, random_seed=SEED)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/site-packages/pymc/sampling/mcmc.py", line 846, in sample
    _mp_sample(**sample_args, **parallel_args)
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/site-packages/pymc/sampling/mcmc.py", line 1243, in _mp_sample
    sampler = ps.ParallelSampler(
              ^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/site-packages/pymc/sampling/parallel.py", line 413, in __init__
    ProcessAdapter(
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/site-packages/pymc/sampling/parallel.py", line 267, in __init__
    self._process.start()
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/context.py", line 301, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/miniconda3/envs/pymc_env/lib/python3.12/multiprocessing/popen_forkserver.py", line 58, in _launch
    f.write(buf.getbuffer())
BrokenPipeError: [Errno 32] Broken pipe

I believe this might be because each job when It finishes its respective job tries to call az.plot_trace() but as other jobs are still using and modifying trace we get the following error.

I imagine the workflow is to do the model fitting in one file, pickle the trace and then load the trace into another script for visual inspection (or use a jupyter notebook which allows for segmentation of cells to make sure all jobs are finished before extra functions are called - Jupyter notebook within vscode - which has been my preferred work around here).

Thanks again for your help, please correct me if any of my conclusions in this comment are wrong. if there is anything else I can provide you with.

9

Ok, to fix your problem you just have to move the plot_trace inside the if __name__… guard.

The reason this happens is that when Python spawns a process it does something called bootstrapping. This just means that each process starts by importing the main module. Python needs to do this so that each child process has access to the functions and modules that were required by main to be able to run. After the import statement finishes, the parent process communicates the job that each child has to do, the child does the work, communicates that it finished with whatever extra info that was requested and terminates. Your spanned processes fail to plot the race because there is no idata variable defined for them while bootstrapping __main__. So when the parent process tries to communicate the task that they have to do, it finds that the communication pipe is broken. The traceback for the cause of the error is also lost for the main process because it never was able to communicate with the child. Depending on where you were running the script, the traceback for the spawned process might be visible in the terminal or not.

1 Like
10

Thanks for that, it does fix the issue.

For future reference would you then suggest coding essentially the entire script within the if __name__ == __main__ guard clause for coding or do you have good tips to know when this clause is needed?

11

That’s more of a Python best practice question. The answer is that it depends on what you’re writing. Is it something that soils work as an importable module, is it just an executable script, or something else? I can point you to this that makes some suggestions.
I personally try to put as little as possible inside the if __name__ stuff. I usually write stuff inside functions and then call those functions in the if __name__. But that’s just how I usually do things and you can try out anything else that works for you.