Small-N Bayesian Poisson regression, sign flip on leave-one-out — sanity check on approach?

Questions v5
,
1

Working on a small applied project: testing whether a technical “readiness” score for e-commerce vendors predicts how often an LLM agent mentions them in purchase-recommendation queries. Poisson regression, mentions ~ Normal-prior alpha + beta * standardized(readiness), PyMC, 4 chains, r_hat ≤ 1.001.

With N=10 vendors: beta = -0.056, 94% HDI [-0.124, 0.011].
Removing the single most-mentioned vendor (N=9): beta = +0.080, HDI [-0.006, 0.167].

Sign flips on leave-one-out with a single observation. Posterior predictive check shows the full-N model underestimates observed variance (looks like overdispersion driven by that one high-count vendor); the N=9 model reproduces it fine.

Questions:

  1. Is a straight Poisson the wrong likelihood here given the overdispersion signal, or is N just too small to tell? Leaning toward “just report both fits and be honest about instability” rather than switching to NegBinomial on 9-10 points — thoughts?
  2. Repetitions (20 per query) aren’t independent draws of the readiness→mentions relationship, they’re repeated samples from the same generative system for a fixed vendor. Right call to keep N = number of unique vendors (not repetitions) as the unit of inference here, or is there a defensible way to use repetition-level variance without pretending it’s independent?
  3. Any obvious red flag in reporting a leave-one-out sensitivity check like this as “the estimate is unstable at this sample size” rather than picking one model and running with it?

Happy to share more context on the applied setup if useful — trying to keep this post focused on the modeling question rather than the domain.

2

  1. N = 10 at this scale is going to be too small to tell much. Your prior’s going to be doing a lot of the inferential lifting no matter what the likelihood is. When you try to fit negative binomials with small data sets, it’s challenging because it’s easy to explain high counts by either a high mean or a high over dispersion, i.e., it reduces identifiability and puts even more pressure on the prior.

  2. If the data is grouped by some factor like vendor ID, you can use a hierarchical model. But that’s really challenging to fit with small N because now there are even more parameters.

  3. Reporting anything from N = 10 is a bit of a red flag already.

That estimator does not look unstable. Flipping the sign is no big deal. It just means the data is compatible with a value of zero. Both those 94% intervals say the distribution is compatible with either sign. With only a highest density interval, we don’t know where the other 6% of the probability mass falls—it could be all above, all below, or a mix.

You can look at the data under bootstrap resampling and estimate the variance of the posterior mean estimator with 10 data points. This evaluates the frequentist coverage of the Bayesian estimator.

Also, I’d recommend reporting 90% or 95% (if you have a lot of draws) central intervals. Anything else is going to require a footnote about why you’re violating standard conventions.

3

Thank you very much for taking the time to provide such a detailed and helpful reply.

The point about HDI versus the central interval is what struck me the

most — I hadn’t realised that ArviZ’s default of 94 per cent concealed

that ambiguity regarding the distribution of the residual tail. I’ll switch straight away

to central percentiles (95 per cent) for the next revision.

The explanation about the sign flip was also helpful: I was interpreting

the instability as a problem with the pipeline, rather than as the expected behaviour

of an effect compatible with zero — that distinction matters a great deal

in terms of how I’m presenting the results.

I’m also implementing the bootstrap you suggested to estimate the

frequentist variance of the estimator — it seems to me to be the right check to provide

a second, independent perspective on stability, in addition to the

sensitivity check I’d already carried out.

If you’re up for it, once I have the corrected version incorporating these changes,

I’d be keen to hear your second opinion — I fully understand if you don’t

have the time, but the quality of this feedback has made me want to give it some thought.

Thanks again,

Alex