# Sampling doesn't start when njobs \> 1 for some models

**URL:** <https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853>\
**Category:** Questions\
**Created:** [February 12, 2018, 6:15am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853 "2018-02-12T06:15:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![aplavin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aplavin/32/259_2.png) [@aplavin](https://discourse.pymc.io/u/aplavin)\
**Post date:** [February 12, 2018, 6:15am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/1 "2018-02-12T06:15:14Z")

</div>

Recently I noticed that I can’t sample certain models with several parallel chains, while others work ok. For a minimal example of what doesn’t work:

```auto
import numpy as np
import pymc3 as pm

n = 96
X = np.random.rand(n, 1)
observed = np.random.rand(n)

with pm.Model() as model:
    cov = pm.gp.cov.ExpQuad(1, 1)
    gp = pm.gp.Latent(mean_func=pm.gp.mean.Zero(), cov_func=cov)
    gp_f = gp.prior('gp_f', X=X)
    val_obs = pm.Normal('val_obs', mu=gp_f, sd=0.1, observed=observed)

    trace = pm.sample(njobs=2)

```

No matter if run in a separate python script or from a notebook, the output is the following:

```auto
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [gp_f_rotated_]
  0%| | 0/1000 [00:00<?, ?it/s]

```

Then it keeps indefinitely in this state, not loading the CPU at all. The traceback when interrupting the process after some waiting:

```auto
Traceback (most recent call last):
  File "tmp.py", line 14, in <module>
    trace = pm.sample(njobs=2)
  File "*/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py", line 437, in sample
    trace = _mp_sample(**sample_args)
  File "*/anaconda3/lib/python3.6/site-packages/pymc3/sampling.py", line 967, in _mp_sample
    traces = Parallel(n_jobs=cores)(jobs)
  File "*/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 789, in __call__
    self.retrieve()
  File "*/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 699, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "*/anaconda3/lib/python3.6/multiprocessing/pool.py", line 638, in get
    self.wait(timeout)
  File "*/anaconda3/lib/python3.6/multiprocessing/pool.py", line 635, in wait
    self._event.wait(timeout)
  File "*/anaconda3/lib/python3.6/threading.py", line 551, in wait
    signaled = self._cond.wait(timeout)
  File "*/anaconda3/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
^C

```

This is really a _minimal_ example: if I reduce `n` from 96 to 95 sampling succeeds! Various other models also work without problems. And the example above works correctly if set `njobs = 1`.

I can reproduce it on two Linux PCs, where everything python-related is installed in the same way using anaconda in a docker container.  
Any ideas how to debug the issue?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [February 12, 2018, 6:25am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/2 "2018-02-12T06:25:06Z")

</div>

This is likely a memory problem. I see it all the time on my mac… Setting njobs=1 helps because either the model can run, and if it fail now it wont fail silently.

---

<div class="post-metadata">

**Author:** ![aplavin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aplavin/32/259_2.png) [@aplavin](https://discourse.pymc.io/u/aplavin)\
**Post date:** [February 12, 2018, 6:28am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/3 "2018-02-12T06:28:11Z")

</div>

By memory problem you mean low memory, right? That’s surprising because the machines have 64 gigs of ram, and anyway, much larger models including GPs can be sampled (using njobs = 1).

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [February 12, 2018, 6:32am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/4 "2018-02-12T06:32:10Z")

</div>

hmm, in that case maybe it is a joblib problem, I remember seeing one in issues: [https://github.com/pymc-devs/pymc3/issues/2640](https://github.com/pymc-devs/pymc3/issues/2640) not sure if it is related though.

---

<div class="post-metadata">

**Author:** ![aplavin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aplavin/32/259_2.png) [@aplavin](https://discourse.pymc.io/u/aplavin)\
**Post date:** [February 12, 2018, 6:37am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/5 "2018-02-12T06:37:58Z")

</div>

It’s very difficult to guess what’s the problem as it doesn’t raise any exception - just stalls indefinitely. Maybe there is something like “verbose” mode in pymc, or similar?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [February 12, 2018, 6:42am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/6 "2018-02-12T06:42:02Z")

</div>

There is a theano verbose mode, but in this case the problem is likely joblib related. Maybe you can try turning on the verbose in joblib [https://pythonhosted.org/joblib/parallel.html](https://pythonhosted.org/joblib/parallel.html)

---

<div class="post-metadata">

**Author:** ![aplavin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aplavin/32/259_2.png) [@aplavin](https://discourse.pymc.io/u/aplavin)\
**Post date:** [February 12, 2018, 6:53am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/7 "2018-02-12T06:53:07Z")

</div>

Well, the `joblib` verbose mode doesn’t seem to help at all: it prints many messages of the form `Pickling array (shape=(*,), dtype=float32).`, and still no errors while sampling stalls.

---

<div class="post-metadata">

**Author:** ![madarshahian](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/madarshahian/32/722_2.png) [@madarshahian](https://discourse.pymc.io/u/madarshahian)\
**Post date:** [May 11, 2018, 8:36pm UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/8 "2018-05-11T20:36:43Z")

</div>

> [@aplavin](#):
>
> models

I am curious if there is any update on this post? I have similar problem.

---

<div class="post-metadata">

**Author:** ![aplavin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aplavin/32/259_2.png) [@aplavin](https://discourse.pymc.io/u/aplavin)\
**Post date:** [May 11, 2018, 8:51pm UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/9 "2018-05-11T20:51:02Z")

</div>

Yeah, I also still experience this issue, and not only for the model mentioned in the first post. Entirely different classes of models don’t sample in multiple processes, so that’s not related to gaussian processes as I thought at first.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [May 11, 2018, 9:58pm UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/10 "2018-05-11T21:58:12Z")

</div>

I suspect it is something to do with joblib pickling - unfortunately cannot really pin down where went wrong yet.

---

<div class="post-metadata">

**Author:** ![magoarcano](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@magoarcano](https://discourse.pymc.io/u/magoarcano)\
**Post date:** [November 3, 2018, 4:16pm UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/11 "2018-11-03T16:16:42Z")

</div>

It worked for me using:  
trace\_g = pm.sample(1100, cores=1)

---

<div class="post-metadata">

**Author:** ![no0ne](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/no0ne/32/2119_2.png) [@no0ne](https://discourse.pymc.io/u/no0ne)\
**Post date:** [September 6, 2019, 2:44am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/12 "2019-09-06T02:44:05Z")

</div>

Was this issue ever resolved? I seem to be experiencing the same problem, but only on linux. I have two identical anaconda environments: one on my desktop running macOS, and the other on a linux box in a rack running ubuntu. If I sample with chains=2 and cores=1, everything runs fine on both machines. If I sample with chains=2 and cores=2, then running on the mac works fine while running on linux results in the sampling sitting at 0% indefinitely until I kill it. I’m not really sure how to begin diagnosing the issue as there is no error, and the python environments are identical as far as I can tell.

---

<div class="post-metadata">

**Author:** ![fbartolic](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fbartolic/32/2757_2.png) [@fbartolic](https://discourse.pymc.io/u/fbartolic)\
**Post date:** [September 6, 2019, 9:59am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/13 "2019-09-06T09:59:05Z")

</div>

I’m experiencing the same issue with certain models on macOS as well…

---

<div class="post-metadata">

**Author:** ![aseyboldt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aseyboldt/32/5795_2.png) [@aseyboldt](https://discourse.pymc.io/u/aseyboldt)\
**Post date:** [September 6, 2019, 1:36pm UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/14 "2019-09-06T13:36:17Z")

</div>

The multicore sampling is completely rewritten by now, so if there still is an issue with frozen chains, it clearly is different. Are you using the most recent pymc3 release? Does this happen sporadically or regularly with one model (if so, which)?

---

<div class="post-metadata">

**Author:** ![kix2](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kix2](https://discourse.pymc.io/u/kix2)\
**Post date:** [January 17, 2020, 9:28am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/15 "2020-01-17T09:28:21Z")

</div>

I also have this issue. I have the latest version, I just installed the module last week. For me basically it used to work on multiple cores and now it only works on one. same model, same code. I’m running on Python 3.7 on a Mac os

---

<div class="post-metadata">

**Author:** ![aseyboldt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aseyboldt/32/5795_2.png) [@aseyboldt](https://discourse.pymc.io/u/aseyboldt)\
**Post date:** [January 17, 2020, 9:40am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/16 "2020-01-17T09:40:46Z")

</div>

Can you share the model that shows this behaviour? Also, what exactly is happening?

---

<div class="post-metadata">

**Author:** ![kix2](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kix2](https://discourse.pymc.io/u/kix2)\
**Post date:** [January 17, 2020, 9:54am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/17 "2020-01-17T09:54:13Z")

</div>

![Screenshot 2020-01-17 at 10.53.12](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/1/1c647f70a373dac6dd1ceb9476021753ea92c736.png)

This is the model. The issue is: until 2 days ago I could sample 4 chains on 8 cores with no issue. Now, if I run it with more than one core, the sampler just doesn’t start.

---

<div class="post-metadata">

**Author:** ![aseyboldt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aseyboldt/32/5795_2.png) [@aseyboldt](https://discourse.pymc.io/u/aseyboldt)\
**Post date:** [January 17, 2020, 9:57am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/18 "2020-01-17T09:57:55Z")

</div>

Can you write a minimal model that I can run myself?  
What happend between now and 2 days ago? Did you update (pymc or something else)? From what version to which? Conda or pip?  
What exactly does “the sampler just doesn’t start” mean? Do you still get the progress bar, but it stays at 0 draws, do you get an error message…

---

<div class="post-metadata">

**Author:** ![kix2](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kix2](https://discourse.pymc.io/u/kix2)\
**Post date:** [January 17, 2020, 10:16am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/19 "2020-01-17T10:16:27Z")

</div>

We can use this notebook I found online, which also doesn’t work: [https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian\_lr/Bayesian%20Linear%20Regression%20Project.ipynb](https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian_lr/Bayesian%20Linear%20Regression%20Project.ipynb) (cells 39-40)

By ‘sampler doesn’t start’ I mean it’s forever frozen at this screen:

 ![Screenshot 2020-01-17 at 11.04.49](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/4/4bdfbe2818a5c98c17018f18eb7921edfa1cd972.png)

So nothing really happened actually. I was running a model on 4 chains, 8 cores, target\_accept=.9 because I was following this tutorial here ([https://docs.pymc.io/notebooks/Diagnosing\_biased\_Inference\_with\_Divergences.html](https://docs.pymc.io/notebooks/Diagnosing_biased_Inference_with_Divergences.html) ), but it was 2 am so I went to sleep after the model finished running, and then the next day (without even restarting the jupyter kernel, it didn’t work anymore - by didn’t work I mean the sampled wouldn’t start like in the image above).

Anyway, I suspected I had some memory issues, or there were other processes running on my computer so I closed everything, restarted everything and at some point attempted reinstalling pymc over pip (just like it was installed originally).

---

<div class="post-metadata">

**Author:** ![kix2](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@kix2](https://discourse.pymc.io/u/kix2)\
**Post date:** [January 17, 2020, 10:48am UTC](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853/20 "2020-01-17T10:48:40Z")

</div>

This is a very random update, but I didn’t want to waste any more of your time. I did a force reinstall of all the packages in my virtual python environment, and now it somehow works. I’m really sorry for commenting here, I was trying to get this to work again for the last 2 days with no avail. I have no idea what happened still. Anyway, thank you for the super quick reply I really appreciate the entire package as I’m trying to learn my way through probabilistic modelling.

[Next page](https://discourse.pymc.io/t/sampling-doesnt-start-when-njobs-1-for-some-models/853.md?page=2)