An error message about 'can't pickle fortran objects'

Hallo
I got a question…
I had my model run before, but today I got the error message.
The only thing I could possible change is that I upgraded the package…I am not sure if it could be related. Does someone have some information about this issue? Thanks a lot.

File “/home/user/Documents/PyMC3/BNN_V1.py”, line 189, in construct_nn_L1
train_trace = pm.sample(draws =5000, tune = 3000)
File “/home/user/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py”, line 442, in sample
trace = _mp_sample(**sample_args)
File “/home/user/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py”, line 982, in _mp_sample
traces = Parallel(n_jobs=cores, mmap_mode=None)(jobs)
File “/home/user/anaconda3/lib/python3.6/site-packages/joblib/parallel.py”, line 789, in call
self.retrieve()
File “/home/user/anaconda3/lib/python3.6/site-packages/joblib/parallel.py”, line 699, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File “/home/user/anaconda3/lib/python3.6/multiprocessing/pool.py”, line 644, in get
raise self._value
File “/home/user/anaconda3/lib/python3.6/multiprocessing/pool.py”, line 424, in _handle_tasks
put(task)
File “/home/user/anaconda3/lib/python3.6/site-packages/joblib/pool.py”, line 371, in send
CustomizablePickler(buffer, self._reducers).dump(obj)
TypeError: can’t pickle fortran objects

Can you try setting cores=1 in pm.sample?

2 Likes

Then it works. But why is that before it is not a problem ? :thinking:

Then this is not related to PyMC3 - it is a joblib problem.
Try upgrading joblib, please report back if you still seeing this problem.

Hallo,
I got this error again even though yesterday it still worked. I already got up-to-date joblib and I also tried with core = 1 but the error msg still showed up.

The message looks like this

Thanks a lot in advance. :slight_smile:

The error is not related to njobs - you are saving the model and trace using pickle right? Not sure why this error occurs but could you try pickle them separately?

yes, that’s what I tried to do.

I have my code now like this

def save_model(train_trace, neural_network):
with open ( ‘file.pkl’, ‘wb’) as buff:
pickle.dump ({‘model’: neural_network}, buff)
pickle.dump ({‘trace’: train_trace}, buff)

But I am not sure if I am doing it right…:confused:

what about saving the trace and model separately like:

with open('model.pkl', 'wb') as buff1:
    pickle.dump({‘model’: neural_network}, buff1)
with open('trace.pkl', 'wb') as buff2:
    pickle.dump({‘trace’: train_trace}, buff2)

Just trying to understand whether the error is from pickling the model or the trace.

Hm…I still got the same error message…:confused: but it’s in the trace part.
image

Is it possible that I changed something in the system when trying doing something else with I/O under python? Or is it possible that I can just make it back to previous status?

This is quite strange as there should be any fortran objects in the trace… and I have never see this error myself before so I am not quite sure how to solve it. Maybe a workaround is to set the trace backend to HDF5 and save the trace directly to the disk.

The model is secretly stored in the trace object, but I am not sure why. You can access it with, for example, trace._straces[0].model. I’ll add this to my list of things to look at when I have more time…

Thanks a lot for all the help.
After trying deleting the theano cache (.theano folder under home directory) and reinstalling pymc3 from master, it still didn’t work…But the same code worked in my another laptop. So I just reinstalled the whole packages including python. And it works now. I guess I changed something behind when I was trying some code related to I/O, but didn’t notice that.

Hi @junpenglao ,

I faced the same error, even though I updated joblib, I still cannot use more than one core.
It only works for cores=1

Please help.
Thanks

3 Likes

Are you on WinOS? We still have persistent WinOS multiple core issue last time I check. @michaelosthege will probably know more.

Hi @junpenglao ,

yes, I am using WinOS. It seems like there are much problems with win OS compare to other OS.
Hopefully we can find a solution to allow it run in WinOS.
I only notice this issue (core=1) problem when I try to solve gaussian problems.
I did not encounter multiple cores issue when I solve other distribution problems.

Thank you very much.

1 Like

@junpenglao @GMCobraz_T @michaelosthege

Seconding this as another WinOS user having issues with the pymc3 install, even after creating a new environment and following the instructions on Git step-by-step.

Seems like WinOS users have trouble 1. using multiprocessing and 2. linking Theano with the proper BLAS setup (e.g. defaults to Numpy C-API implementation)

A separate but related thread that sites the same “can’t pickle fortran objects” issue links to a website explaining how users can install Ubuntu in their WinOS environment and use pymc3 that way.

That might work for some people, but ideally a fix for WinOS would be better.

1 Like

Hello everyone, Is there any update on this?
I am facing a similar issue to @brianleblanc and @GMCobraz_T while trying to fit a Gaussian Process with core>1 in an anaconda environment.

Windows 10 OS
PyMC3 3.11.4
joblib 0.17

1 Like

Bumping this - running into the same issue with a GP model.

Pymc 3.11.4, joblib 1.1.0

This probably popped up in other threads but the issue extends to MvNormal.

1 Like