For some of the models (coin flip, exponential, hyperbolic) running pm.loo(trace, model) does not give any warnings at all.
For others (e.g. HyperboloidA) I get
UserWarning: Estimated shape parameter of Pareto distribution is
greater than 0.7 for one or more samples.
You should consider using a more robust model, this is because
importance sampling is less likely to work well if the marginal
posterior and LOO posterior are very different. This is more likely to
happen with a non-robust model and highly influential observations.
happen with a non-robust model and highly influential observations.""")
That does seem to coincide with the The estimated number of effective samples is smaller than 200 for some parameters. warning from the sampling step.
I re-ran with 10,000 samples and still get both errors. Seems a bit odd that effective sample size is <200 given 10,000 samples. Chains are visually fine, Rhat is fine, posterior predictive check confirms a sensible fit. Not quite sure how to progress.
Here’s the model, sitting inside a small class:
Thanks for the help on this by the way