# Correctly specifying a model with a multimodal distribution

**URL:** <https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405>\
**Category:** version agnostic\
**Tags:** modeling\
**Created:** [May 11, 2022, 11:07pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405 "2022-05-11T23:07:10Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![matsuo\_basho](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@matsuo\_basho](https://discourse.pymc.io/u/matsuo_basho)\
**Post date:** [May 11, 2022, 11:07pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/1 "2022-05-11T23:07:10Z")

</div>

Hello PyMC3 community.

I’m trying to build a model based on a multimodal disribution.

I have the following code, but am not sure that it’s correct:

1. The way I’m specifying priors is using the .dist syntax, which makes me think it’s not stochastic.
2. If I plot the prior predictive and posterior predictive samples, they are not very different from each other.

```auto
import numpy as np
import pymc3 as pm
import matplotlib.pyplot as plt

W = np.array([0.5, 0.5, 0.5])

data = np.array([1.73, 1.59, 1.64, 1.58])

with pm.Model() as model_mix:
    # Mixing variable
    w = pm.Dirichlet('w', np.ones_like(W))

    dist1 = pm.Normal.dist(mu = 1.72, sigma = 0.01)
    dist2 = pm.Normal.dist(mu = 1.78, sigma = 0.01)
    dist3 = pm.StudentT.dist(mu = 1.88, sigma = 0.01, nu = 5)

    like = pm.Mixture('like', w=w, comp_dists=[dist1, dist2, dist3], observed=data)

    trace = pm.sample(2000, tune = 8000, cores=1)

params_prior_mix = pm.sample_prior_predictive(model = model_mix, samples = 2000)
params_pred_mix = pm.sample_posterior_predictive(trace, 1000, model_mix)
predictions_vec_mix = params_pred_mix['like'].reshape(4000)

plt.hist(np.random.choice(predictions_vec_mix, 1000), label='Posterior', color='r')
plt.hist(params_prior_mix['like'][:,0], label='Prior')
plt.hist(data, color='pink', label='Measured Distribution')
plt.legend()    

```

Do I need to model the distributions of the parameters of dist1, dist2, dist3 in order to correctly specify the priors?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 11, 2022, 11:17pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/2 "2022-05-11T23:17:36Z")

</div>

Have you checked [this notebook](https://docs.pymc.io/projects/examples/en/latest/mixture_models/gaussian_mixture_model.html)?

Regarding your prior and posterior samples looking similar, you have very little data, so that may be what’s expected

---

<div class="post-metadata">

**Author:** ![matsuo\_basho](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@matsuo\_basho](https://discourse.pymc.io/u/matsuo_basho)\
**Post date:** [May 11, 2022, 11:57pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/3 "2022-05-11T23:57:02Z")

</div>

I haven’t seen this, very useful! The article refers to pymc4 but pymc3 also has the NormalMixture method. I tried is using pymc3 and it’s very slow with the sampling. Is NormalMixture optimized in pymc4 to run faster, by chance?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 12, 2022, 12:02am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/4 "2022-05-12T00:02:42Z")

</div>

The pymc3 version of that notebooks is [here](https://docs.pymc.io/en/v3/pymc-examples/examples/mixture_models/gaussian_mixture_model.html). Sampling categorical models like these tend to be difficult due to the categorical/discrete nature of some of the parameters. However, as you can see by comparing those 2 notebooks, there have been changes in v4. Whether they help speed-wise, I’m not sure. But it may be worth trying both.

---

<div class="post-metadata">

**Author:** ![matsuo\_basho](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@matsuo\_basho](https://discourse.pymc.io/u/matsuo_basho)\
**Post date:** [May 12, 2022, 12:10am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/5 "2022-05-12T00:10:11Z")

</div>

I see, the pymc4 version is considerably simpler. The generated data clearly has 3 modes, but it appears that the sigma prior has the same parametes (halfnormal distr, sigma=1).

So I think the idea here to just use a very weak prior? In my case, since my data is scarce, I would want to specify differing sigmas for each of the prior distributions in the mixture. How would I do that?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 12, 2022, 12:15am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/6 "2022-05-12T00:15:03Z")

</div>

You may be able to just pass in a vector of `sigma` values when defining \sigma. Not 100% of that, but you could sample from the prior to check.

---

<div class="post-metadata">

**Author:** ![matsuo\_basho](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@matsuo\_basho](https://discourse.pymc.io/u/matsuo_basho)\
**Post date:** [May 12, 2022, 3:15pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/7 "2022-05-12T15:15:07Z")

</div>

So I tried the following:

```auto
data = np.array([1.73, 1.59, 1.64, 1.58])

with pm.Model(coords={"cluster": range(k)}) as model_mix:
    μ = pm.Normal(
        "μ",
        mu=0,
        sigma=.01,
        transform=pm.distributions.transforms.ordered,
        initval=[1.72, 1.78, 1.88],
        dims="cluster",
    )
    σ = pm.HalfNormal("σ", sigma=np.array([.025, .025, 0.01]), dims="cluster")
    weights = pm.Dirichlet("w", np.ones(k), dims="cluster")
    pred = pm.NormalMixture("x", w=weights, mu=μ, sigma=σ, observed=data)

    trace_mix = pm.sample()

with model_mix:
    trace_mix.extend(pm.sample_prior_predictive())
    trace_mix.extend(pm.sample_posterior_predictive(trace_mix))

plt.hist(trace_mix['prior_predictive']['x'].to_numpy().reshape(-1))

```

However, sampling from the prior predictive as above just gives a normal-like distribution with a mean of 0. So something isn’t right.

Curious what the difference is between the formulation above using NormalMixture versus the example from [here](https://docs.pymc.io/en/v3/api/distributions/mixture.html#pymc3.distributions.mixture.NormalMixture).

I would probably have to specify 2 priors for each of the distributions under this formulation (so 6 priors in total):

```auto
# 2-Mixture Poisson using iterable of distributions.
with pm.Model() as model:
    lam1 = pm.Exponential('lam1', lam=1)
    lam2 = pm.Exponential('lam2', lam=1)

    pois1 = pm.Poisson.dist(mu=lam1)
    pois2 = pm.Poisson.dist(mu=lam2)

    w = pm.Dirichlet('w', a=np.array([1, 1]))

    like = pm.Mixture('like', w=w, comp_dists = [pois1, pois2], observed=data)

```

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 12, 2022, 4:19pm UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/8 "2022-05-12T16:19:40Z")

</div>

> [@matsuo\_basho](#):
>
> However, sampling from the prior predictive as above just gives a normal-like distribution with a mean of 0. So something isn’t right.

Given that the prior on the means of all three components are zero, isn’t this what one would expect?

---

<div class="post-metadata">

**Author:** ![matsuo\_basho](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@matsuo\_basho](https://discourse.pymc.io/u/matsuo_basho)\
**Post date:** [May 14, 2022, 12:53am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/9 "2022-05-14T00:53:17Z")

</div>

So using the formulation from the first link with an array of 3 different sigmas worked. How do I think about the 2 different ways of specifying the mixture models - the pm.NormalMixture versus pm.Mixture. Are they alternate ways of specifying the same thing (when I specify a normal distrs with pm.Mixture) or is there a fundamental difference between them?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 14, 2022, 1:28am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/10 "2022-05-14T01:28:54Z")

</div>

My understanding is that `NormalMixture` and `MixtureSameFamily` are convenience classes to simplify common use-cases, though there could be slight tweaks available in these special cases.

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [May 14, 2022, 7:44am UTC](https://discourse.pymc.io/t/correctly-specifying-a-model-with-a-multimodal-distribution/9405/11 "2022-05-14T07:44:26Z")

</div>

`pm.NormalMixture` simply uses `pm.Mixture` under the hood