Need help for calculating LOO-CV with multiple observed values

Questions
1

Hi there everyone,

I am trying to use loo-cv on my model but I get a lot of k-values above .7 and the warning:
" Estimated shape parameter of Pareto distribution is greater than 0.7 for one or more samples. You should consider using a more robust model, this is because importance sampling is less likely to work well if the marginal posterior and LOO posterior are very different. This is more likely to happen with a non-robust model and highly influential observations."
grafik

For the background: I have built a Bayesian brain model of an experiment in which participants have to carry out hand movements in a restricted timeframe. Their movement is displayed on screen and sometimes it is delayed. The experimental task is to detect these delays.

So basically participants have an expectation (M) about when they get information from vision and proprioception. There are hyperparameters for M as to express that the person usually knows when they are moving.
timepoint when the hand is displayed on screen–> observed variable no. 1
timepoint when they hand starts moving → observed variable no. 2

With this information they infer how big the delay was. Using a GLM approach a logit for the Bernoulli likelihood is calculated which yields a “yes” or a “no delay” response.
response → observed variable no. 3
The model is still a bit more complicated than this but I hope this suffices to explain my problem. Below I have put a depiction of my Bayes net. Sorry if it’s a bit hard to understand.

At the moment my questions are:
Is loo-cv even the right tool for my problem?
Can I fix the problem with the high k-values? What could I do instead?
For Stan there is loo_moment_match which I thought might help but from my research pymc3 does not have such a function.

Any help would be greatly appreciated!!

So now I want to evaluate my model fit using

grafik
grafik1342Ă—659 53.6 KB

1 Like
2

I don’t see why not from what you explained. However, I am not completely clear about what do you want to predict, only the yes/no answer? Or the times too? LOO-CV estimates the predictive accuracy of your model, so defining what are the predictions is key to estimating if they are accurate or not.

It looks like you have one delta_t value per value of response which might be too flexible. In hierarchical models, having very little observations (or a single observation) on a group tends to make those observations “highly influential” and therefore PSIS can’t estimate the cross validation result. That is because in such cases, the full posterior generally differs significantly from the posterior after removing one highly influential observation and the PSIS approximation assumes they are similar, otherwise it is not reliable.

Note that I don’t know the data nor the model and for example delta_t is also constrained by observed_v and response also depends on y and z, so the model might not be “too flexible” (aka non-robust) but I thought it was worth mentioning this as an avenue to get PSIS to work.

We don’t yet have moment matching in ArviZ, only reloo which would mean >60 refits I think in your case but if you are interested it would be great to add it and we’d be happy to help you add this feature.

1 Like