NUTS Sampler: Effective samples is smaller than 200 for some parameters

Questions
1

Hello,

‘The estimated number of effective samples is smaller than 200 for some parameters.’ notification appears when I conduct NUTS sampler. I tried to change ‘mu’ and ‘sd’ values in order to solve the issue, but the output values are highly effected by them.

I looked for previous related topics, however couldn’t find a solution. What could be the possible reason and consequences of it?

with pm.Model() as cohen:
    
    # response

    μ0 = pm.Normal('μ0', mu=0, sd=10, shape=groups)
    σ0 = pm.HalfNormal('σ0', sd=10, shape=groups)
    
    y0 = pm.Normal('y0', mu=μ0[idx], sd=σ0[idx], observed=rsp)   
    
    # reaction time    
    
    μ1 = pm.Normal('μ1', mu=0, sd=10, shape=groups)
    σ1 = pm.HalfNormal('σ1', sd=10, shape=groups)
    
    y1 = pm.Normal('y1', mu=μ1[idx], sd=σ1[idx], observed=rt)
   
    trace_cohen = pm.sample(target_accept=0.99)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [σ1, μ1, σ0, μ0]
Sampling 4 chains, 0 divergences: 100%|████████████████████████████████████████| 4000/4000 [00:27<00:00, 143.73draws/s]
The estimated number of effective samples is smaller than 200 for some parameters.
2

https://discourse.pymc.io/t/the-number-of-effective-samples-is-smaller-than-25-for-some-parameters

Directly quoting from the above post, a low number of effective samples is usually an indication of strong autocorrelation in the chain. You can plot the trace autocorrelation or look at the trace plot directly to check if the sampler is either getting stuck or not exploring enough. Usually, this could be improved by using a more efficient sampler like NUTS, but here you are already using NUTS. Have you tried varying target_accept to be slightly lower, say .95, and/or increasing the number of tuning/sampling? E.g., does pm.sample(5000,tune=15000,target_accept=0.95) also produce <200 effective samples? If all that fails, could it be that you lack a suitable number of observations for this model? You could try mocking up some fake data so that you can try running the model with more observations, to see if that’s a possible explanation. You can also try using more informed priors. My initial concern looking at this is that your model seems to be running super fast, finishing in only 27 seconds.

Also note that PyMC3 might be too sensitive with their effective sample size warnings. STAN’s warning for effective sample size only shows when the n_eff for a parameter is less than 1% of the sample size.

Sorry I’m not helpful with specific advice, but this is what I’ve come across when trying to diagnose this kind of warning before.

1 Like
3

Thank you for the informative response.

I think I will try with increase the number of tuning/sampling and more informative prior. But I cannot pick lower target accept. It gives me some divergences. Also, number of observations are really low right now. This might be the main reason model fails in certain occasions (non-informative priors, lower target accept, autocorrelation etc.).

4

Hi,
Yeah I’d definitely do prior predictive checks and try more informative priors – here is good guidance.
Right now, with mu_0 for instance, you’re basically assuming that 95% of the probability mass for the mean will be between -20 and 20 (2 stds) – this is probably huge, depending on the scale of your data.
Hope this helps :vulcan_salute:

1 Like
5

Thank you very much for the guidance :slightly_smiling_face:

I will try to pick different priors according to the scale and run the model again!

1 Like