Very nice.
In previous work (not in MCMC but just in approximating distributions) I’ve used the MultivariateNormalDiagPlusLowRank (hey, I didn’t name it…) from tensorflow-probability, which (cf here) uses a matrix perturbation like
diag(sig) + VV^T for a (n, k) matrix V with k << n.
In so doing; V is not unique, but there’s no longer the cost of ensuring orthogonal columns. For mass matrix adaptation is an orthogonal system UDU^T strictly required?