Thanks @junpenglao ! Understand you don’t have time to dig, but could you try to provide a bit more detail from what you know off the top of your head (without digging deep into math or the code)?
-
“Yes this is normal sampling from a mixture. There is no good solution in high dimension unfortunately.”
My input space in this toy example only has 3 locations (in one spatial dimension), so it’s not really high dimension. Unless you’re talking the size of the model (# of Thetas)? I don’t understand why the sampler would not have the trace for those “extra” Thetas (without measurements) be 60% of one component and 40% of the other, instead of what looks like 99.9% of one component and 0.1% of the other. -
“NUTS will just adapt to the local geometry and explore one local mode”
This must be the core issue, but I don’t understand what this means. Can a different sampler do this in pymc3, other than NUTS? Or can you suggest if I can try a different package altogether (Stan?), although I really prefer to stay with pymc3 since I’m comfortable in it! -
What I’m trying to do basically is show that as you incorporate more and more measurements of the overall population (Thetas) – ie, as you gain more and more knowledge of the mixture weights and sub-population means – that the uncertainty of the “next” (N+1) Theta (with no measurements) drops down more and more until it essentially approaches the limit, which would be like perfect knowledge of the mixture.
Thanks again!