Gaining intuitions for why sampling fails

Questions
1

I’m back to working with pymc3 again after a couple of years where I haven’t had the chance to use it, so that is great. But, I’m still a noob when it comes to this and I feel that I get stuck on the same issues I had when I last used it, and that is most models that are more advanced than the basic tutorial stuff on real data always have some issues with the sampling (divergent transitions, low number of effective samples, acceptance probability does not match…). For hierachical glm’s the parameterising using the non-centered parameterisation is awesome and solves a lot of problems, but unfortunately not all problems.

So my first question is sort of a general one, how do you go about fixing sampling issues? Are there good tutorials for how to work around common issues in common models? I’ve found quite a lot of material explaining what pathological issues might occur while sampling, but apart from the re-parameterisation mentioned above not a lot on how to actually fix them, but maybe that is a sign on that it is not possible to give general advice?

However, if we want to get more specific, I’m currently trying to implement the Rolling regression model (https://docs.pymc.io/notebooks/GLM-rolling-regression.html) on my own data. So the model looks like:

with pm.Model() as model_randomwalk:
sigma_alpha = pm.Exponential('sigma_alpha', lam=5.)
sigma_beta = pm.Exponential('sigma_beta', lam=5.)

alpha = pm.GaussianRandomWalk('alpha', sigma=sigma_alpha, shape=len(df))
beta = pm.GaussianRandomWalk('beta', sigma=sigma_beta, shape=len(df))
sigma = pm.HalfNormal('sigma', sigma=1)

regression = alpha + beta*df['za']
likelihood = pm.Normal('y', mu=regression, sigma=sigma, observed=df['zs'])
trace_rw = pm.sample(5000, tune=5000, target_accept=.99)

The data is here if anyone want to play with it https://drive.google.com/file/d/1KIEk5NrK7jR5q-uR-tcQExJGxcd_FfCO/view?usp=sharing

My sigma_alpha and sigma_beta parameters does not seem sample well (n effective samples <100, etc). I’ve tried different number of (tuning) steps, different target_accept, different priors, but nothing seems to help. Any tips for what I can do to fix this? Or should I accept that given this data this model is not possible to fit?

2

Glad to see you working on it again. Sampling failures are something I have been struggling to intuitively grasp over time, but I have helped build it over time by searching over many threads in these forums using some filters.

One reason sampling fails that I do grasp intuitively, is when a parameter is being sampled over an impossible region for it, like when sampling a sigma over negative values.

Other reasons for divergences/failures, are a bit more complicated to understand, and I realize now I should be working on that understanding myself.

3

Hi there!
Thanks for this thread, I’ll follow it closely because I have a similar experience :laughing:

Just from my point of view, what worked best in practice were prior predictive checks and regularizing priors – thinking about your priors and see their consequences in graphs. This can really make your model sample instead of crashing (especially with GLMs, where the link function distorts the relationship between the parameters and outcome spaces).
In your case though, the priors for sigma_alpha and sigma_beta look quite good. I’d maybe try reducing the lambda to 1 instead of 5, but maybe you already tried that? In any case, prior pred checks could be valuable here.

Another very good trick is to standardize the predictors (df['za']here). This often simplifies the geometry and really helps sampling as a result.
A final note would be to think hard about how your predictors interact (multicollinearity, confounding, etc.). I don’t have actionable advice here though, as I’m still trying to make sense of it myself and gain some understanding of how this affects sampling.

I know what I’m saying here is not revolutionary but hopefully that will help you or others :slight_smile:

All that being said, I also think some problems can’t be solved quickly: often this stuff is hard and you have to try and fail in order to understand and, in the end, gain some intuition. It can be frustrating and it definitely takes time! But I think it’s worth it – or at least hope so :stuck_out_tongue_winking_eye:

1 Like