# Error when comparing ADVI models with az.compare

**URL:** <https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123>\
**Category:** v5\
**Created:** [August 10, 2022, 11:50pm UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123 "2022-08-10T23:50:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArturoBell](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/arturobell/32/5428_2.png) [@ArturoBell](https://discourse.pymc.io/u/ArturoBell)\
**Post date:** [August 10, 2022, 11:50pm UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123/1 "2022-08-10T23:50:21Z")

</div>

Hi everyone!

I just encountered an error when using az.compare to, well, compare two models fit using ADVI; however, I get the following error:

> TypeError: Encountered error in ic computation of compare.

I used the samples generated from the fit as traces. Should I be using another object, or is it not possible to compare models fit with ADVI?

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![aloctavodia](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/aloctavodia/32/8642_2.png) [@aloctavodia](https://discourse.pymc.io/u/aloctavodia)\
**Post date:** [August 11, 2022, 11:14am UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123/2 "2022-08-11T11:14:32Z")

</div>

Hi,

I found the same issue recently (pymc 4.x). I find out that the computation of the log likelihood is disable on purpose and thus not part of the InferenceData (@ferrine probably knows the reason).

You can still ask pymc to compute the log likelihood for you by doing

trace = mean\_field.sample(1000, return\_inferencedata=False)  
idata = pm.to\_inference\_data(trace=trace, model=model, log\_likelihood=True)

Then you can use that idata to compute LOO.  
While this will get the values computed I will be careful about the results until getting more info from @ferrine or others knowing why it is currently disable by default.

---

<div class="post-metadata">

**Author:** ![ArturoBell](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/arturobell/32/5428_2.png) [@ArturoBell](https://discourse.pymc.io/u/ArturoBell)\
**Post date:** [August 11, 2022, 11:48pm UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123/3 "2022-08-11T23:48:32Z")

</div>

Hi @aloctavodia!

Thank you very much for the reply. I’ll check it out and see if my results make sense, although I will not make any decision until we get more information.

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [August 15, 2022, 1:21pm UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123/4 "2022-08-15T13:21:09Z")

</div>

I think the comparison is not fair. Consider the case when your approximate posterior converged to have narrow high density region, while the true posterior has a broader shape. In the comparison, the narrow model would always win which is not true. I’m not against of returning the log likelihood, but this leads to confusion in interpretation

---

<div class="post-metadata">

**Author:** ![schmaus](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/schmaus/32/5591_2.png) [@schmaus](https://discourse.pymc.io/u/schmaus)\
**Post date:** [September 17, 2022, 4:31pm UTC](https://discourse.pymc.io/t/error-when-comparing-advi-models-with-az-compare/10123/5 "2022-09-17T16:31:56Z")

</div>

@ferrine If I understand it correctly this paper argues for using PSIS loo cross validation for identifying problems with ADVI approximations. They cite the R loo package, but I believe this is the same as the arviz.loo, which in turn uses the `log_likelihood` component from the `InferenceData`.

> **[Yes, but Did It Work?: Evaluating Variational Inference](https://arxiv.org/abs/1802.02538)**
>
> While it's always possible to compute a variational approximation to a
> posterior distribution, it can be difficult to discover problems with this
> approximation. We propose two diagnostic algorithms to alleviate this problem.
> The Pareto-smoothed...