# Mixture with multiple observations

**URL:** <https://discourse.pymc.io/t/mixture-with-multiple-observations/642>\
**Category:** Questions\
**Created:** [December 8, 2017, 11:29pm UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642 "2017-12-08T23:29:56Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@chris](https://discourse.pymc.io/u/chris)\
**Post date:** [December 8, 2017, 11:29pm UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/1 "2017-12-08T23:29:56Z")

</div>

Hello, I’m trying to figure out the right way to generate model for this data in pymc3:

![2017-12-08 15_21_22-Clipboard](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/1X/72bc0fe01ca36c8522ac648459798f0c535969f4.png)

So I have about 70 observations, and only 1 of them is informative. I have a mix of 2 classes which have different informative observations.

I’m able to capture the important observation when it’s only 1 (not a mixture) with the code below. Can anybody give me a hint how to consider it for a mixture? I don’t need a detailed solution, even a hint would be so helpful!

```
with pm.Model() as model:
    
    #pm.GLM.from_formula('y ~ x', y)
    mu_a = pm.Normal('mu_a', mu=0., sd=100.**2)
    sigma_a = pm.HalfCauchy('sigma_a', 5.)
    
    a = pm.Normal('a', mu=mu_a, sd=sigma_a, shape=C)
    
    #MODEL ERROR
    eps = pm.HalfCauchy('eps',5)
    
    y_est = pm.Deterministic('y_est',a)
    
    data_likelihood = pm.Normal('data_likelihood', mu=y_est, sd=eps, observed=y)

    mode = pm.find_MAP()
    print(mode)
    
    trace = pm.sample(draws=1000, tune=200)

pm.traceplot(trace)

```

[example\_data.csv](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/1X/8a274203e89d64040bbf1205a49226bb080d9672.csv) (365.7 KB)

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [December 9, 2017, 8:59am UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/2 "2017-12-09T08:59:09Z")

</div>

This reminds me of [another post here](https://discourse.pymc.io/t/gaussian-mixture-of-regression/537)

You can have a look at the notebook [https://gist.github.com/junpenglao/1907bf019906c125f63126ec4bf23880#file-mixture\_discourse-ipynb](https://gist.github.com/junpenglao/1907bf019906c125f63126ec4bf23880#file-mixture_discourse-ipynb)

I think extending the model above into a softmax should solve the problem.

---

<div class="post-metadata">

**Author:** ![chris](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@chris](https://discourse.pymc.io/u/chris)\
**Post date:** [December 12, 2017, 3:42pm UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/3 "2017-12-12T15:42:28Z")

</div>

Thanks @junpenglao! This is a huge help and I’m able to run my example based off the non-marginalized mixture. I’m still working on figuring out how the marginalized mixture works because the performance involving categorical variables is pretty brutal.

---

<div class="post-metadata">

**Author:** ![chris](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@chris](https://discourse.pymc.io/u/chris)\
**Post date:** [December 13, 2017, 12:04am UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/4 "2017-12-13T00:04:46Z")

</div>

Hi @junpenglao. I’m trying to extend your notebook to the multivariate case that I have but I can’t seem to get the dimensions correct. Can I still use the pm.NormalMixture function when I have multiple observations like in the picture above? I’m not sure where to apply softmax because my output data is a 2-D array and not a vector of classes.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [December 13, 2017, 6:31am UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/5 "2017-12-13T06:31:13Z")

</div>

handling shape in `pm.NormalMixture` might be tricky, maybe try reshaping the output data into a vector (also the `mu` and `sd` for the mixture)

---

<div class="post-metadata">

**Author:** ![chris](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@chris](https://discourse.pymc.io/u/chris)\
**Post date:** [December 14, 2017, 12:46am UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/6 "2017-12-14T00:46:48Z")

</div>

I have something close that’s giving my best result yet. It’s using a DensityDist like the example from [Gaussian Mixture Model with ADVI](http://docs.pymc.io/notebooks/gaussian-mixture-model-advi.html) in the PyMC3 docs. I’d like to try your idea of applying a softmax constraint, but I’m not sure where to do that. Is that an additional factor in the “logps” variable? I have a [github notebook](https://github.com/christacular/pymc3_applications/blob/master/PyMC3_Multivariate_Mixture_Model.ipynb) modified to use my data if you’d like to take a peek :).

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [December 14, 2017, 5:58am UTC](https://discourse.pymc.io/t/mixture-with-multiple-observations/642/7 "2017-12-14T05:58:23Z")

</div>

Hi @chris,  
Actually, I don’t think my suggestion of using a softmax is well thought. Looking at the data generation process, each row is not necessary sum to 1 right? If you have `K` \> 2 you can try modeling the `pi` using a softmax, but I am not sure it is better than using Dirichlet (which use a stick-breaking process internally).