Is it possible to carry out multiprocess sampling without using Pickle? My PYmC3 model requires the use of an FEM solver tool (esys-escript) to calculate the output and whenever i set njobs>1 or cores>1 i get the following error:
Multiprocess sampling (2 chains in 2 jobs)
CompoundStep
Slice: [pr_vec]
Slice: [sigma_log__]
Traceback (most recent call last):
File “reducedMCMC7.py”, line 107, in
trace = pm.sample(step=pm.Slice(), start=start, njobs=2)
File “/home/nick/.local/lib/python2.7/site-packages/pymc3/sampling.py”, line 442, in sample
trace = _mp_sample(**sample_args)
File “/home/nick/.local/lib/python2.7/site-packages/pymc3/sampling.py”, line 982, in _mp_sample
traces = Parallel(n_jobs=cores, mmap_mode=None)(jobs)
File “/home/nick/.local/lib/python2.7/site-packages/joblib/parallel.py”, line 789, in call
self.retrieve()
File “/home/nick/.local/lib/python2.7/site-packages/joblib/parallel.py”, line 699, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File “/usr/lib/python2.7/multiprocessing/pool.py”, line 572, in get
raise self._value
RuntimeError: Pickling of “esys.escriptcore.escriptcpp.Data” instances is not enabled (http://www.boost.org/libs/python/doc/v2/pickle.html)
I’ve read that dill is more robust than pickle but i’m not sure how to implement it. Can dill be used instead of Pickle? Alternatively, is there some other way to prevent PYmC3 from using Pickle but still carry out multiprocess sampling?
Cheers.