Since we’re assuming one mode per variable, this seems very HMM to me. Use a low within-trace switch probability, and a higher cross-trace switch probability, and treat the emission distribution as normal wiht unknown mean and variance.
But (implementing in pymc3 and) using an HMM to solve an identifiability issue seems like total overkill. Is there a way to parametrize your space or loadings so that the posterior is proper?