In my notebook, what I did is to have the MvNormal model the latent variable, and have the predictor as the observation of said latent variable. The reason to do so (instead of assigning the observation directly to the MvNormal) is that it allows me to do center vs noncenter parameterization. I also tried your way of modeling it, but it does not work very well (havent figure it out why yet).
I think you might find the discussion in Cholesky decomposition and correlation among random effects between me and @Jack_Caster useful.