Need help for calculating LOO-CV with multiple observed values

I don’t see why not from what you explained. However, I am not completely clear about what do you want to predict, only the yes/no answer? Or the times too? LOO-CV estimates the predictive accuracy of your model, so defining what are the predictions is key to estimating if they are accurate or not.

It looks like you have one delta_t value per value of response which might be too flexible. In hierarchical models, having very little observations (or a single observation) on a group tends to make those observations “highly influential” and therefore PSIS can’t estimate the cross validation result. That is because in such cases, the full posterior generally differs significantly from the posterior after removing one highly influential observation and the PSIS approximation assumes they are similar, otherwise it is not reliable.

Note that I don’t know the data nor the model and for example delta_t is also constrained by observed_v and response also depends on y and z, so the model might not be “too flexible” (aka non-robust) but I thought it was worth mentioning this as an avenue to get PSIS to work.

We don’t yet have moment matching in ArviZ, only reloo which would mean >60 refits I think in your case but if you are interested it would be great to add it and we’d be happy to help you add this feature.

1 Like