Error when comparing ADVI models with az.compare

Hi everyone!

I just encountered an error when using az.compare to, well, compare two models fit using ADVI; however, I get the following error:

TypeError: Encountered error in ic computation of compare.

I used the samples generated from the fit as traces. Should I be using another object, or is it not possible to compare models fit with ADVI?

Thank you in advance!

1 Like

Hi,

I found the same issue recently (pymc 4.x). I find out that the computation of the log likelihood is disable on purpose and thus not part of the InferenceData (@ferrine probably knows the reason).

You can still ask pymc to compute the log likelihood for you by doing

trace = mean_field.sample(1000, return_inferencedata=False)
idata = pm.to_inference_data(trace=trace, model=model, log_likelihood=True)

Then you can use that idata to compute LOO.
While this will get the values computed I will be careful about the results until getting more info from @ferrine or others knowing why it is currently disable by default.

4 Likes

Hi @aloctavodia!

Thank you very much for the reply. I’ll check it out and see if my results make sense, although I will not make any decision until we get more information.

2 Likes

I think the comparison is not fair. Consider the case when your approximate posterior converged to have narrow high density region, while the true posterior has a broader shape. In the comparison, the narrow model would always win which is not true. I’m not against of returning the log likelihood, but this leads to confusion in interpretation

@ferrine If I understand it correctly this paper argues for using PSIS loo cross validation for identifying problems with ADVI approximations. They cite the R loo package, but I believe this is the same as the arviz.loo, which in turn uses the log_likelihood component from the InferenceData.