Numerical Issues with StickBreaking in ADVI

[junpenglao]

@Dominik

I do not quit get it. lambda x: softmax(StickBreacking().backward(x)) is an isomorphism Rn−1→Δn−1\mathbb R^{n-1}\rightarrow\Delta^{n-1}

It’s not - it just that in your case the Normal(0, 1) regulated it, if you set to components = pm.Normal('components', mu=0., sigma=100., shape=(nd, nd)) you can see quite clearly the problem introduce by the softmax. Also, note that the softmax is performed on the components, there is no StickBreacking().backward(x).