# Why does \`transform=pm.distributions.transforms.ordered\` lead to worse convergence?

**URL:** <https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944>\
**Category:** Questions\
**Created:** [August 25, 2021, 10:51pm UTC](https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944 "2021-08-25T22:51:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pawlu](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pawlu/32/4391_2.png) [@pawlu](https://discourse.pymc.io/u/pawlu)\
**Post date:** [August 25, 2021, 10:51pm UTC](https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944/1 "2021-08-25T22:51:46Z")

</div>

Similar to how a gaussian distribution is fit to a trendline in Bayesian linear regression, I’m attempting to fit a gaussian mixture to a trend line.

I’m having some issues with convergence specifically on the mixture parameters.

As expected for me in a mixture model, I initially get what appears to be some label switching.

I attempt to break symmetry via `pm.distributions.transforms.ordered` on the means (the line commented out on the code) but this seems to add new peaks to the trace plot.

Why is `pm.distributions.transforms.ordered` causing these new peaks to appear?

```python
import numpy as np
import pymc3 as pm
import theano.tensor as tt

data = ... # pandas DataFrame containing data
model_usage_data = data[["scaled_value"]]
model_day = model_usage_data.index.to_numpy().reshape(-1, 1)
coords = {"day": model_usage_data.index}
groups = 2

with pm.Model(coords=coords) as model:
    alpha = pm.Normal("alpha", mu=0, sd=10)
    beta = pm.Normal("beta", mu=0, sd=1)

    day_data = pm.Data("day_data", model_day)
    broadcast_day = tt.concatenate([day_data, day_data], axis=1)

    trend = pm.Deterministic("trend", alpha + beta * broadcast_day)

    _means = pm.Normal(
        "_means",
        mu=[[0, 0.1]],
        sd=10,
        shape=(1, groups),
        # Will be toggling this line
        # transform=pm.distributions.transforms.ordered,
        testval=np.array([[0, 0.2]]),
    )
    means = pm.Deterministic("means", _means + trend)

    p = pm.Dirichlet("p", a=np.ones(groups))
    sds = pm.HalfNormal("sd", sd=10, shape=groups)

    pm.NormalMixture("y", w=p, mu=means, sd=sds, observed=model_usage_data)

    trace = pm.sample(
        draws=draws,
        tune=tune,
        target_accept=0.90,
        max_treedepth=15,
        return_inferencedata=False,
    )

```

### Without `transform=pm.distributions.transforms.ordered`

 ![download](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/f/fcb64f49344e446438d7731a0e29d5200c374b75.jpeg)

### With `transform=pm.distributions.transforms.ordered`

 ![download](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/7/7f9b54d8493e546efc4204308d25d1e10ce1e555.jpeg)

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [August 27, 2021, 8:13am UTC](https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944/2 "2021-08-27T08:13:03Z")

</div>

Just a wild guess but perhaps the Ordered transform is failing because of the 2D shape? Does the same happen if you do something like:

```python
_means = pm.Normal(
        "_means",
        mu=[0, 0.1],
        sd=10,
        shape=groups,
        # Will be toggling this line
        # transform=pm.distributions.transforms.ordered,
        testval=np.array([0, 0.2]),
    )
    means = pm.Deterministic("means", tt.reshape(_means, (1, groups)) + trend)

```

Another guess is that one of your `_means` might be redundant with the `alpha` (they seem to both work as intercepts) and somehow adding the ordering transform makes this redundancy even more salient. In that case removing `alpha` should help.

---

<div class="post-metadata">

**Author:** ![pawlu](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pawlu/32/4391_2.png) [@pawlu](https://discourse.pymc.io/u/pawlu)\
**Post date:** [August 27, 2021, 8:24pm UTC](https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944/3 "2021-08-27T20:24:32Z")

</div>

> Another guess is that one of your `_means` might be redundant with the `alpha` (they seem to both work as intercepts)

Good catch! Can’t believe I missed that 😅.

Also good idea on reshaping after ordering. But it doesn’t seem to help in this case.

New model, without `alpha`, and reshaping after. Again I’ll be toggling the commented out line for both traces.

```python
import numpy as np
import pymc3 as pm
import theano.tensor as tt

data = ... # pandas DataFrame containing data
model_usage_data = data[["scaled_value"]]
model_day = model_usage_data.index.to_numpy().reshape(-1, 1)
coords = {"day": model_usage_data.index}
groups = 2

with pm.Model(coords=coords) as model:
    beta = pm.Normal("beta", mu=0, sd=1)

    day_data = pm.Data("day_data", model_day)
    broadcast_day = tt.concatenate([day_data, day_data], axis=1)

    trend = pm.Deterministic("trend", beta * broadcast_day)

    _means = pm.Normal(
        "_means",
        mu=[0, 0.1],
        sd=10,
        shape=(1, groups),
        # Will be toggling this line
        # transform=pm.distributions.transforms.ordered,
        testval=np.array([0, 0.2]),
    )
    means = pm.Deterministic(
        "means", tt.reshape(_means, (1, groups)) + trend
    )

    p = pm.Dirichlet("p", a=np.ones(groups))
    sds = pm.HalfNormal("sd", sd=10, shape=groups)

    pm.NormalMixture("y", w=p, mu=means, sd=sds, observed=model_usage_data)

    trace = pm.sample(
        draws=draws,
        tune=tune,
        target_accept=0.90,
        max_treedepth=15,
        return_inferencedata=False,
    )

```

### Without `transform=pm.distributions.transforms.ordered`

 ![download](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/9/90b2bbead41114dc5925efb28a3f7aec289a2bbe.png)

### With `transform=pm.distributions.transforms.ordered`

 ![download (1)](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/7/7e44076cf45d1ba897ab4e8f3c683ff026301711.png)

---

<div class="post-metadata">

**Author:** ![pawlu](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pawlu/32/4391_2.png) [@pawlu](https://discourse.pymc.io/u/pawlu)\
**Post date:** [August 27, 2021, 8:29pm UTC](https://discourse.pymc.io/t/why-does-transform-pm-distributions-transforms-ordered-lead-to-worse-convergence/7944/4 "2021-08-27T20:29:30Z")

</div>

Even before reshaping (plots in original post), it does seem like `transform=pm.distributions.transforms.ordered` is doing its job—the label switching in `_means` goes away.

It’s just the affect on all the other variables that’s the problem.