# Is there an example on how to work with generalized mixture models?

**URL:** <https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726>\
**Category:** Questions\
**Created:** [January 12, 2018, 9:03pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726 "2018-01-12T21:03:05Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![ericmjl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ericmjl/32/196_2.png) [@ericmjl](https://discourse.pymc.io/u/ericmjl)\
**Post date:** [January 12, 2018, 9:03pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/1 "2018-01-12T21:03:05Z")

</div>

I am working on a simplest complex example to record my learning journey here.

> <https://github.com/ericmjl/bayesian-analysis-recipes/blob/master/mixture-model.ipynb>

Working with Normal Mixtures works fine, but I’m not quite sure how to debug the error that I’m getting at the bottom of the notebook when trying to do generalized Mixtures. Am I setting up the model incorrectly?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [January 12, 2018, 9:59pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/2 "2018-01-12T21:59:42Z")

</div>

The correct syntax is changing:  
`components = pm.Poisson('components', mu=lam, shape=(2,))`  
to  
`components = pm.Poisson.dist(mu=lam, shape=(2,))`

Also, careful of label switching in this case. More information see [https://gist.github.com/junpenglao/4d65d1a9bf80e8d371446fadda9deb7a](https://gist.github.com/junpenglao/4d65d1a9bf80e8d371446fadda9deb7a)

---

<div class="post-metadata">

**Author:** ![ericmjl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ericmjl/32/196_2.png) [@ericmjl](https://discourse.pymc.io/u/ericmjl)\
**Post date:** [January 14, 2018, 3:53am UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/3 "2018-01-14T03:53:05Z")

</div>

Thanks @junpenglao! I must have missed the `.dist(...)` thing somewhere. Was this documented in the docs? Perhaps I can put in a PR to show how this syntax gets used?

Btw, I got the Poisson Mixture working! Thanks for your help! 😄

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [January 14, 2018, 8:10am UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/4 "2018-01-14T08:10:30Z")

</div>

I think it is in the doc somewhere but you are right, it is not very clear from the docstring. It would be great to add an Example into the docstring like what you are adding for Normal etc!

---

<div class="post-metadata">

**Author:** ![ericmjl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ericmjl/32/196_2.png) [@ericmjl](https://discourse.pymc.io/u/ericmjl)\
**Post date:** [January 17, 2018, 3:51pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/5 "2018-01-17T15:51:47Z")

</div>

Thanks @junpenglao! One thing I’m finding is I’m getting errors with putting together Weibull mixtures. Might you have an intuition as to what’s happening? I updated my `mixture-model` notebook ([https://github.com/ericmjl/bayesian-analysis-recipes/blob/master/notebooks/mixture-model.ipynb](https://github.com/ericmjl/bayesian-analysis-recipes/blob/master/notebooks/mixture-model.ipynb)), and it can be found at the bottom.

The error message is two-fold:

- No modes.
- Object not iterable.

Neither makes sense to me, as I’m following the pattern above for the Poisson mixtures.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [January 17, 2018, 4:49pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/6 "2018-01-17T16:49:45Z")

</div>

It might not work directly out of the box… There are some problem with the mixture model in terms of dimension etc, for example:

> [@Mixture of two beta geometric with beta shape \> 1](https://discourse.pymc.io/t/mixture-of-two-beta-geometric-with-beta-shape-1/636):
>
> I am trying to model a simple mixture of beta geometric distributions (following up from my previous question: [Recovering parameters of long tailed Beta-Geometric data](https://discourse.pymc.io/t/recovering-parameters-of-long-tailed-beta-geometric-data/601)), but I get this error: ValueError: Input dimension mis-match. (input[0].shape[0] = 2, input[1].shape[0] = 500) my attempt this time: import pymc3 as pm model = pm.Model() with model: switch = pm.Uniform('switch', 0, 1) a1 = pm.Uniform('a1', 0.0001, 1) a2 = pm.Uniform('a2', 0.0001, 1) b1 = pm.Uniform('b1',…

> <https://github.com/pymc-devs/pymc/issues/2478>
>
> I'm trying to create a mixture of multinomials using PyMC using \`pm.Mixture\` con…struct as described in \[documentation\](https://pymc-devs.github.io/pymc3/api/distributions/mixture.html). However, I'm running into dimension mismatch error:
> \`\`\`
> T = 4
> V = 10
> with pm.Model() as model:
> H = pm.Dirichlet("H", a=np.ones(V), shape=V)
> phi = \[pm.Multinomial('phi\_%s' %t, n=10, p=H, shape=V) for t in range(T)\]
> phi\_dists = \[comp.distribution for comp in phi\]
> G0 = pm.Mixture('G0', w = np.ones(T)/T, comp\_dists = phi\_dists)
> \`\`\`
> This code results in the following error:
> \`ValueError: Input dimension mis-match. (input\[0\].shape\[1\] = 4, input\[1\].shape\[1\] = 10)\`
> I do have a mixture with 4 components each of size 10.
> 
> I also tried constructing the mixture manually:
> \`\`\`
> with model:
> phi\_mix = \[w\[t\]\*phi\_top\[t\] for t in range(T)\]
> G0 = pm.Deterministic('G0', sum(phi\_mix))
> \`\`\`
> However, the above code lead to a different downstream error. So I was wondering how I can construct a mixture of multinomials using PyMC3? This should be a common task for discrete data models.
> This question is also posted on \[stack overflow\](https://stackoverflow.com/questions/45495241/mixture-of-multinomials-in-pymc3).

> <https://github.com/pymc-devs/pymc/issues/2614>
>
> \## Description of your problem
> From 
> https://stackoverflow.com/questions/46581…700/pymc3-sampling-multiple-numbers-from-normal-mixture-distribution-doesnt-work
> 
> See the example below
> 
> \*\*Please provide a minimal, self-contained, and reproducible example.\*\*
> 
> import pymc3 as pm
> 
> pm.NormalMixture.dist(
> w=0.5 \* np.ones(2),
> mu=np.ones(2),
> sd=np.ones(2)).random(size=2)
> 
> 
> \*\*Please provide the full traceback.\*\*
> 
> \`\`\`
> \---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> \<ipython-input-25-bccf91a988a4\> in \<module\>()
> 2 w=0.5 \* np.ones(2),
> 3 mu=np.ones(2),
> \----\> 4 sd=np.ones(2)).random(size=2)
> 
> ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/mixture.py in random(self, point, size, repeat)
> 133 dist\_shape=self.shape,
> 134 size=size).squeeze()
> \--\> 135 comp\_samples = self.\_comp\_samples(point=point, size=size, repeat=repeat)
> 136 
> 137 if comp\_samples.ndim \> 1:
> 
> ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/mixture.py in \_comp\_samples(self, point, size, repeat)
> 101 def \_comp\_samples(self, point=None, size=None, repeat=None):
> 102 try:
> \--\> 103 samples = self.comp\_dists.random(point=point, size=size, repeat=repeat)
> 104 except AttributeError:
> 105 samples = np.column\_stack(\[comp\_dist.random(point=point, size=size, repeat=repeat)
> 
> ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/continuous.py in random(self, point, size, repeat)
> 260 return generate\_samples(stats.norm.rvs, loc=mu, scale=tau\*\*-0.5,
> 261 dist\_shape=self.shape,
> \--\> 262 size=size)
> 263 
> 264 def logp(self, value):
> 
> ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/distribution.py in generate\_samples(generator, \*args, \*\*kwargs)
> 389 prefix\_shape,
> 390 \*args, \*\*kwargs)
> \--\> 391 return reshape\_sampled(samples, size, dist\_shape)
> 
> ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/distribution.py in reshape\_sampled(sampled, size, dist\_shape)
> 305 
> 306 if np.size(sampled) == 1 or repeat\_shape or dist\_shape:
> \--\> 307 return np.reshape(sampled, repeat\_shape + dist\_shape)
> 308 else:
> 309 return sampled
> 
> ~/.venv/sampled/lib/python3.6/site-packages/numpy/core/fromnumeric.py in reshape(a, newshape, order)
> 230 \[5, 6\]\])
> 231 """
> \--\> 232 return \_wrapfunc(a, 'reshape', newshape, order=order)
> 233 
> 234 
> 
> ~/.venv/sampled/lib/python3.6/site-packages/numpy/core/fromnumeric.py in \_wrapfunc(obj, method, \*args, \*\*kwds)
> 55 def \_wrapfunc(obj, method, \*args, \*\*kwds):
> 56 try:
> \---\> 57 return getattr(obj, method)(\*args, \*\*kwds)
> 58 
> 59 # An AttributeError occurs if the object does not have
> 
> ValueError: cannot reshape array of size 4 into shape (2,)
> \`\`\`
> 
> \*\*Please provide any additional information below.\*\*
> It looks like we are using the same \`size\` parameter in two places where they might mean different things \[around here\](https://github.com/pymc-devs/pymc3/blob/master/pymc3/distributions/mixture.py#L130):
> 
> w\_samples = generate\_samples(random\_choice,                               
> w=w,                                         
> broadcast\_shape=w.shape\[:-1\] or (1,),        
> dist\_shape=self.shape,
> size=size).squeeze()
> comp\_samples = self.\_comp\_samples(point=point, size=size, repeat=repeat)  
> 
> \## Versions and main components
> 
> \* PyMC3 Version: \`master\`
> \* Theano Version: 0.9.0
> \* Python Version: 3.6
> \* Operating system: osx
> \* How did you install PyMC3: (conda/pip) \`pip\`

There are some work needs to be done there…

---

<div class="post-metadata">

**Author:** ![ericmjl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ericmjl/32/196_2.png) [@ericmjl](https://discourse.pymc.io/u/ericmjl)\
**Post date:** [January 17, 2018, 5:04pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/7 "2018-01-17T17:04:48Z")

</div>

I noticed that the Weibull distribution doesn’t have a mode defined. Is that because of the definition of the Weibull mode being conditionally dependent on the value of the alpha parameter? ([https://en.wikipedia.org/wiki/Weibull\_distribution](https://en.wikipedia.org/wiki/Weibull_distribution))

Can we have conditional modes in PyMC3 distributions, or do they have to be analytical, non-conditional modes?

---

<div class="post-metadata">

**Author:** ![fabianmoss](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fabianmoss/32/1618_2.png) [@fabianmoss](https://discourse.pymc.io/u/fabianmoss)\
**Post date:** [March 14, 2019, 11:30am UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/8 "2019-03-14T11:30:13Z")

</div>

Thanks for the great notebook @ericmjl! It was a great help for me. I tried to extend it with other mixtures, e.g. binomial but could not get it to work.

My artificial data is

```auto
comp1 = np.random.binomial(n=7, p=.1, size=(1000,)) # - 2
comp2 = np.random.binomial(n=7, p=.7, size=(500,)) # + 4

mix = np.concatenate([comp1, comp2])

```

and my model speficitation

```auto
with pm.Model() as binomial:
    n = pm.Geometric('n', p=1/7)
    p = pm.Uniform('p', lower=.5, upper=1.)
    w = pm.Dirichlet('w', a=np.array([1,1]))
    
    comps = pm.Binomial.dist(n=n, p=p, shape=(2,))
    
    mixture = pm.Mixture('mixture', comp_dists=comps, w=w, observed=mix)

```

However, when I try to sample from the model,

```auto
with binomial:
    trace = pm.sample(1000, chains=2, cores=2)

```

my notebook crashes. What am I doing wrong here?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 14, 2019, 2:01pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/9 "2019-03-14T14:01:11Z")

</div>

could be a memory problem - could you paste the crash message from terminal?  
Also, try:

```python
with binomial:
    trace = pm.sample(1000, chains=2, cores=1)

```

---

<div class="post-metadata">

**Author:** ![fabianmoss](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fabianmoss/32/1618_2.png) [@fabianmoss](https://discourse.pymc.io/u/fabianmoss)\
**Post date:** [March 14, 2019, 2:54pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/10 "2019-03-14T14:54:01Z")

</div>

The error message I get is:

```auto
You can find the C code in this temporary file: C:\Users\FABIAN~1\AppData\Local \Temp\theano_compilation_error_z64rcaov
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function _module.py", line 1082, in _constructor_Function
    f = maker.create(input_storage, trustme=True)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function _module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\link.py", li ne 699, in make_thunk
    storage_map=storage_map)[:3]
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\vm.py", line 1091, in make_all
    impl=impl))
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line 955, in make_thunk
    no_recycling)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 1189, in module_from_key
    module = lnk.compile_cmodule(location)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1523, in compile_cmodule
    preargs=preargs)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2396, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('The following error happened while compiling the node', Softmax(El emwise{sub,no_inplace}.0), '\n', 'Compilation failed (return status=3): ', '[So ftmax(<TensorType(float64, matrix)>)]')
forrtl: error (200): program aborting due to control-C event

```

To be clear: I did not press CTRL+C 🙂

I tried your suggestion and it kind of worked. Could you explain me why?

Also, for `binomial.check_test_point()` I get

```auto
n -1.95
p_interval__ -2.41
w_stickbreaking__ -1.39
mixture NaN
Name: Log-probability of test_point, dtype: float64
Name: Log-probability of test_point, dtype: float64

```

The result looks odd though:

 ![traceplot](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/9/9b684fc1700064faed5976bcfd5638aa2c431de9.png)

---

<div class="post-metadata">

**Author:** ![fabianmoss](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fabianmoss/32/1618_2.png) [@fabianmoss](https://discourse.pymc.io/u/fabianmoss)\
**Post date:** [March 14, 2019, 3:05pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/11 "2019-03-14T15:05:51Z")

</div>

If I replace `n` with a constant value in `pm.Binomial.dist` I don’t get a NaN value for `mixture`.

---

<div class="post-metadata">

**Author:** ![lucianopaz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/lucianopaz/32/2186_2.png) [@lucianopaz](https://discourse.pymc.io/u/lucianopaz)\
**Post date:** [March 14, 2019, 4:53pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/12 "2019-03-14T16:53:12Z")

</div>

Are you on windows?

---

<div class="post-metadata">

**Author:** ![fabianmoss](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fabianmoss/32/1618_2.png) [@fabianmoss](https://discourse.pymc.io/u/fabianmoss)\
**Post date:** [March 14, 2019, 5:05pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/13 "2019-03-14T17:05:57Z")

</div>

Yes I am.

---

<div class="post-metadata">

**Author:** ![lucianopaz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/lucianopaz/32/2186_2.png) [@lucianopaz](https://discourse.pymc.io/u/lucianopaz)\
**Post date:** [March 14, 2019, 9:35pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/14 "2019-03-14T21:35:34Z")

</div>

There is a known recurrent issue that happens on windows when `sample` is run with multiple cores. The problem is not actually restricted to windows. Unix OSes don’t show it so often because the multiprocessing backend defaults to `fork`, which behaves better than `spawn` and `forkserver` which are the available backends on windows.

The errors are very diverse but your particular issue looks strongly related to [this](https://github.com/pymc-devs/pymc3/issues/3140) and [this](https://github.com/Theano/Theano/issues/6678) issues. It is clearly a `theano` related problem in which the spawned process for some reason that we don’t understand yet, fails to compile the model’s `logp` (what actually happens is that it tries to unpickle a `theano.function` and to do so, it tries to recompile it in the new process). The only work around we have for now is to run `sample` with `cores=1`.

---

<div class="post-metadata">

**Author:** ![fabianmoss](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/fabianmoss/32/1618_2.png) [@fabianmoss](https://discourse.pymc.io/u/fabianmoss)\
**Post date:** [March 16, 2019, 5:08pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/15 "2019-03-16T17:08:25Z")

</div>

Thanks a lot! I will check out the issues you mention. Is there anything I can test to help figuring out what’s going wrong with `sample` on my machine? Also, is this likely to disappear with PyMC4 being based on TensorFlow?

---

<div class="post-metadata">

**Author:** ![lucianopaz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/lucianopaz/32/2186_2.png) [@lucianopaz](https://discourse.pymc.io/u/lucianopaz)\
**Post date:** [March 16, 2019, 9:25pm UTC](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726/16 "2019-03-16T21:25:04Z")

</div>

I think that the particular problem with multiprocessing will disappear in pymc4, because we could rely on tensorflow itself to distribute the runs across many devices. We are still far from addressing this, however.  
You are, of course, more than welcome to try to explore the root cause of your problematic model. I never managed to reproduce those errors, and I honestly don’t know what is going wrong on the theano side.