Any experience with reLOO?

I notice that ArViZ has a reloo function but I haven’t found any PyMC3 examples using it. Has anyone had success with this? I know its flagged as “experimental” right now.

I want to compare models that have 1-2 data points with LOO pareto shapes > 0.7. It would be great to calculate these LOO point without PSIS.

1 Like

It would be great if you could help testing it out. Right now there is only one example available for PyStan on the website but trying it out on real models would be amazing.

I have recently refocused on this a little with hopes to expanding to PSIS-LFO and SMC, and I started working on the PyMC3 wrapper too on this PR. It would be great if you could checkout the branch and try it. I’ll rebase it with master.

Getting it to work with PyMC3 has its issues, but it can be sorted out. The PR actually has 2 proposals that try to abstract the process as much as possible, feedback on the general API and on how these 2 alternatives compare will be most appreciated. I still want to try a 3rd option using symbolic_pymc though, I hope I’ll have time within the following 2 weeks.

I personally prefer the alternative that relies more on xarray instead of the one that uses pymc3 more intensively as I think it requires less mental overhead to use. Here is a link to the notebook which I think you’ll be able to use modifying only the sel_observations method.

The link to the second alternative is here. Unlike in the previous case, here there is no need to rewrite with numpy the pointwise log likelihood formula, but the resulting class may be slightly harder to generalize.

Let me know if you need help adapting this to your models

2 Likes

Also, the wrapper is currently thought to recompile the model every time a refit is needed because when the shape of the observations is changed the model needs to be recompiled. With LFO every refit has changes on the shape of the observations but for reloo this is not the case, if your model were really slow to compile we could look into this but at least one recompilation will be needed. With 1-2 datapoints to which reloo needs to be applied it may not be worth it to go to all this trouble.

There is an example of reloo usage with ArviZ now on one issue: https://github.com/arviz-devs/arviz/issues/1404. I though it could be interesting to people reading this.

2 Likes

I’ve looked through your examples a couple of times now, and I’m still lost. Can’t figure out how to actually use reloo.

What kind of model are you trying to use? Are the issues creating the sampling wrapper or using the wrapper to call reloo?