Arviz Compare -TypeError: Encountered error in ELPD computation of compare

Questions version agnostic
1

Hi folks;
I am getting an error(as below) while trying to compare my pymc models on Google Colab. It seems like I am getting same error on pymc v5 on my local machine. I haven’t tested this code on other pymc versions; Did I mis-specify anything here?

import pymc as pm
import arviz as az

with pm.Model() as m1:
    a = pm.Normal('a',0,1)
    sigma =pm.Exponential('sigma',1)
    mu=a
    y_obs=pm.Normal("y_obs",mu=mu,sigma=sigma,observed=[0,1,23])
    trace1 = pm.sample(5000,cores=2)

with pm.Model() as m2:
    a = pm.Normal('a',0,10)
    sigma =pm.Exponential('sigma',1)
    mu=a
    y_obs=pm.Normal("y_obs",mu=mu,sigma=sigma,observed=[0,1,23])
    trace2 = pm.sample(5000,cores=2)

az.compare(
    {'m1':trace1,'m2':trace2},
    scale='deviance',
    ic='waic',
    method="BB-pseudo-BMA"
)

When I run this code I get an Error as below>>

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/arviz/stats/stats.py in _calculate_ics(compare_dict, scale, ic, var_name)
    447             try:
--> 448                 compare_dict[name] = ic_func(
    449                     convert_to_inference_data(dataset),

5 frames
TypeError: log likelihood not found in inference data object

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
TypeError: Encountered error trying to compute waic from model m1.

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/arviz/stats/stats.py in compare(compare_dict, ic, method, b_samples, alpha, seed, scale, var_name)
    177         (ics_dict, scale, ic) = _calculate_ics(compare_dict, scale=scale, ic=ic, var_name=var_name)
    178     except Exception as e:
--> 179         raise e.__class__("Encountered error in ELPD computation of compare.") from e
    180     names = list(ics_dict.keys())
    181     if ic == "loo":

TypeError: Encountered error in ELPD computation of compare.
2

PyMC’s default used to be to include the pointwise log-probabilities in the InferenceData object returned by pm.sample. That was changed at some point (it’s quite slow and memory intensive if you don’t need it), so now you have to ask for it. If you add idata_kwargs={'log_likelihood':True} to pm.sample in all your models, you will be able to use az.compare.

3 Likes
3

It worked, thanks

4

I just ran into this issue when trying to use compare. I already ran several traces and would like to avoid having to re-run them. Is it possible to compute the log likelihood on saved traces?

5

You want pm.compute_log_likeihood. Usage looks like this:

with model:
    idata = pm.compute_log_likelihood(idata, extend_inferencedata=True)
2 Likes