Jacobian adjustment

Questions
1

When I browse the Stan forum, I often read about the need for adjusting the Jacobian. However, I rarely find this topic on this forum, and I am wondering if we should instead be more careful.

From what I understood, in general, we need need to adjust the Jacobian when we perform a change of variable. From the Stan manual (p. 291):

[there is a] difference between a change of variables and a simple
variable transformation. A transformation samples a parameter, then transforms it,
whereas a change of variables transforms a parameter, then samples it. Only the
latter requires a Jacobian adjustment

I have hard time to understand when we would need to adjust the Jacobian in our PyMC3 code:

  • If I read the quote correctly, the use of Deterministic in PyMC3 to transform a parameter should be a safe operation, right? Are there cases in which this is not true?
  • Stan allows to declare something like log(y) ~ normal(mu, sigma), that is we can apply a function to the left hand side of the statement (This seems to be the typical example that need a Jacobian adjustment.) PyMC3, however, does not allow such statement, right? If so, we will not encounter such problem in our code.
2 Likes
2

My take is that it depends on where you are applying the transformation.

In the latent/parameter space, transformation like center/non-center parameterization usually has a neglectable impact on the parameter estimation. Although there would be small biases depending on your estimator, but if you sample from the model and use samples to compute posterior mean you are usually safe (see Stan case study by Bob Carpenter, pymc3 port here).

But if your transformation is done on the observed (eg, demeaning and scaling the observation), then you need to be much more careful. For example, see this notebook with a simple example using linear regression. We dont allow things like log(y) ~ normal(mu, sigma), and we dont recommend to have transformation done on observations (actually I was surprised that it is allowed, we dont have any example on the website doing that).

An additional sidenote is that, reparameterization will always change your logp, and if you account for the jacobian, it would defeat the purpose of reparameterization. Here is a long demo using Neal’s funnel: https://github.com/junpenglao/All-that-likelihood-with-PyMC3/blob/master/Notebooks/Neals_funnel.ipynb

3 Likes