General cookbook for reparametrization

The need for reparametrization is quite a common problem I suppose.

I read the nice paper by Betuncourt, Girolami(2013) which deals with funnels for hierarchical models and I think I understand now that the general problem for Euclidian HMC (and other samplers excluding Riemannian HMC) are strong local (on small scales) correlations which significantly differ from global correlations on different, typically larger scales. These features of parameter space are typically called “funnels”.
I understand that a common reparametrization choice for hierarchical models it to go from centered to non-centered parametrizations. But what about a more general case which also features funnels?

In section 12.1 of the Baysian Data Analysis book Gelman seems to suggest a more general approach of parameter expansion through the addition of auxiliary parameters. The selection of additional parameters are guided by the conditional distributions of parameters between each other, if some dependency could induce strong shrinkage, it is offset (or multiplied, i.e. offset in logarithmic space) by some independent auxiliary parameter. This is mostly useful for the Gibbs sampler though.

Could you recommend some resources or common practice how to reparametrize towards reducing funnels in general?

1 Like

The paper of Betancourt and Girolami is my go to reference as well. I am not aware of a general treatment/cookbook to archive best parameterization. My understanding is that it dependents on your data and model, and sometimes non-center works better but you find cases where centered parameterization works better as well - I think I saw somewhere that non-center is better when there’s not much data, but beyond that you need to try different thing and see which one is better.