The number of effective samples is smaller than 25% for some parameters

Questions
1

I’m trying to run the code in chapter 1 of the book that Thomas Wiecki recommended: Probabilistic Programming and Bayesian Methods for Hackers.
However, I get the error: “The number of effective samples is smaller than 25% for some parameters.”

The data being analyzed is the number of text messages sent per day for a period of 70 days.

with pm.Model() as model:
    alpha = 1.0/count_data.mean()  
    lambda_1 = pm.Exponential("lambda_1", alpha)    
    lambda_2 = pm.Exponential("lambda_2", alpha)    
    
    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data - 1)
    
    idx = np.arange(n_count_data) 
    lambda_ = pm.math.switch(tau > idx, lambda_1, lambda_2)

    observation = pm.Poisson("obs", lambda_, observed=count_data) 

    step = pm.Metropolis()                       
    trace = pm.sample(10000, tune=5000,step=step)

What does this error mean exactly, and why am I getting it? I tried changing sample numbers but that didn’t help.

2 Likes
2

The low number of effective samples is usually an indication of strong autocorrelation in the chain. You can plot the trace autocorrelation or look at the trace plot directly.

Usually, this could be improved by using more efficient sampler like NUTS. However, in this case since you are mixing discrete random variable with continuous variables, sampling is likely going to be a bit difficult.

You can find more information about effective sample size from Gelman et al’s book Bayesian Data Analysis. The stan language manual has some detail information on how they are computed.

I think the warning threshold in PyMC3 is currently a bit too sensitive (for example, in Stan they only output warning when < .01%). But big thumbs up to you for actually read the warning and try to improve your model!!

4 Likes
3

Thanks junpenglao. Your comments were really helpful. Now I have 2 more books to read :slight_smile:
I modeled it again without assigning ‘step’, so PyMC3 can automatically assign an appropriate sampler:

with model:
    trace = pm.sample(10000, tune=5000)

As a result, pymc3 used two samplers:

NUTS: [lambda_2_log__, lambda_1_log__]
Metropolis: [tau]

I still got the same warning of effective sample < 25%. I’m assuming this is due to the Metropolis sampler as opposed to the NUTS. So i continued reading…
This post (correlation - What is causing autocorrelation in MCMC sampler? - Cross Validated) mentions: “when using MH algorithms, to some extent you can reduce or increase your autocorrelations by adjusting the step size of proposal distribution”.

I looked at the pymc3 documentation but couldn’t find ways to reduce step size for Metropolis (https://docs.pymc.io/api/inference.html#module-pymc3.step_methods.metropolis). How can I do that? Do you think that would help?

2 Likes
4

The statement from StackExchange is only partially right - you reduce or increase the probability of accepting the proposal at each step by adjusting the step size, and too large or too small step size both lead to high autocorrelation - small step size you are likely to accept all step which leads the chain to display random walk behaviour, large step size you are likely to reject all step which leads the chain to stuck. And both case the autocorrelation is high. This is why the step size is tune automatically in PyMC3 and in general we dont recommend tuning step size by hand, unless you know exactly what you are doing.

So in your case, the first thing to do is check the trace plot, see if the chain is random walking or stuck.

In PyMC3, you dont control the step size directly except some simple cases (here could be one). You can however control the acceptance probability - large acceptance probability means the sampler will be tuned so that the step size is smaller.

2 Likes
5

thank you