Parallel sampling without pickle in version 4.0.0b3

I’m using a black-box likelihood and run into an error when parallel sampling. version 4.0.0b3

Traceback (most recent call last):
  File "/miniconda3/envs/rjmcmc/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/miniconda3/envs/rjmcmc/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/.vscode-oss/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 331, in run_module
    run_module_as_main(target_as_str, alter_argv=True)
  File "/miniconda3/envs/rjmcmc/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/miniconda3/envs/rjmcmc/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Documents/education/5.eth_ma/thesis/code/model_framework/sketches/blasi_model_selection.py", line 264, in <module>
    trace = pm.sample(ndraws, tune = nburn, step = step, discard_tuned_samples = True, cores=2, chains=nchains)
  File "/miniconda3/envs/rjmcmc/lib/python3.9/site-packages/pymc/sampling.py", line 569, in sample
    mtrace = _mp_sample(**sample_args, **parallel_args)
  File "/miniconda3/envs/rjmcmc/lib/python3.9/site-packages/pymc/sampling.py", line 1493, in _mp_sample
    sampler = ps.ParallelSampler(
  File "/miniconda3/envs/rjmcmc/lib/python3.9/site-packages/pymc/parallel_sampling.py", line 411, in __init__
    step_method_pickled = cloudpickle.dumps(step_method, protocol=-1)
  File "/miniconda3/envs/rjmcmc/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/miniconda3/envs/rjmcmc/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 602, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle 'SwigPyObject' object

I read here that it was possible to sample without using pickle now. But I can’t seem to figure out how to turn this on.

1 Like

Looking at the discussion of this error elsewhere, it sounds like the error may be misleading and caused by other problems arising in one of the jobs/threads (e.g., the blackbox likelihood?). What happens when you sample on a single core?

Oh single core works fine. I initially switched to cores=1 to set this problem aside. But I’m starting to need the performance now.

Interesting. I guess the other suggestion at this point would be to try the beta of PyMC v4, if for no other reason than it would indicate where the problem might lie. Otherwise, I might create an issue and see if anyone has any clues. The thread you linked to is now several years old and much has changed in the interim.

Well I’m currently on 4.0.0b3.
So this seems to be a bug and I’m not missing some kind of option to turn off pickling ?

Sorry, totally missed the fact that you are already on v4. I can’t confirm whether it’s a bug or not, but raising an issue would probably get you an answer.

alright I’ll do that then. Thanks for the help !