Can't convert trace to Arviz

Questions
1

I’m trying to convert a trace to arviz but I’m getting an exception.

trace = pymc3.backends.sqlite.load(file)
converter = arviz.data.io_pymc3.PyMC3Converter(                                                                                                                                                                 
        trace=trace,                                                                                                                                                                                            
        model=model,                                                                                                                                                                                            
)     

In [7]: converter.to_inference_data()


In [7]: converter.to_inference_data()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/tools/miniconda3/lib/python3.7/site-packages/pymc3/backends/base.py in __getitem__(self, idx)
    139         try:
--> 140             return self.point(int(idx))
    141         except (ValueError, TypeError):  # Passed variable or variable name.

~/tools/miniconda3/lib/python3.7/site-packages/pymc3/backends/sqlite.py in point(self, idx)
    268             values = _rows_to_ndarray(self.db.cursor)
--> 269             var_values[varname] = values.reshape(self.var_shapes[varname])
    270         return var_values

ValueError: cannot reshape array of size 0 into shape (6,)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/data/r4/projects/wellbeing/notebooks/state_year_load.py in <module>
----> 1 converter.to_inference_data()

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/io_pymc3.py in to_inference_data(self)
    366             "posterior": self.posterior_to_xarray(),
    367             "sample_stats": self.sample_stats_to_xarray(),
--> 368             "log_likelihood": self.log_likelihood_to_xarray(),
    369             "posterior_predictive": self.posterior_predictive_to_xarray(),
    370             "predictions": self.predictions_to_xarray(),

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/base.py in wrapped(cls, *args, **kwargs)
     35                 if all([getattr(cls, prop_i) is None for prop_i in prop]):
     36                     return None
---> 37             return func(cls, *args, **kwargs)
     38
     39         return wrapped

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/base.py in wrapped(cls, *args, **kwargs)
     35                 if all([getattr(cls, prop_i) is None for prop_i in prop]):
     36                     return None
---> 37             return func(cls, *args, **kwargs)
     38
     39         return wrapped

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/io_pymc3.py in log_likelihood_to_xarray(self)
    212             return None
    213         try:
--> 214             data = self._extract_log_likelihood()
    215         except TypeError:
    216             warnings.warn(

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/base.py in wrapped(cls, *args, **kwargs)
     35                 if all([getattr(cls, prop_i) is None for prop_i in prop]):
     36                     return None
---> 37             return func(cls, *args, **kwargs)
     38
     39         return wrapped

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/base.py in wrapped(cls, *args, **kwargs)
     35                 if all([getattr(cls, prop_i) is None for prop_i in prop]):
     36                     return None
---> 37             return func(cls, *args, **kwargs)
     38
     39         return wrapped

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/io_pymc3.py in _extract_log_likelihood(self)
    177                 log_like_chain = [
    178                     self.log_likelihood_vals_point(point, var, log_like_fun)
--> 179                     for point in self.trace.points([chain])
    180                 ]
    181                 log_likelihood_dict.insert(var.name, np.stack(log_like_chain), chain)

~/tools/miniconda3/lib/python3.7/site-packages/arviz/data/io_pymc3.py in <listcomp>(.0)
    176             for chain in self.trace.chains:
    177                 log_like_chain = [
--> 178                     self.log_likelihood_vals_point(point, var, log_like_fun)
    179                     for point in self.trace.points([chain])
    180                 ]

~/tools/miniconda3/lib/python3.7/site-packages/pymc3/backends/base.py in __getitem__(self, idx)
    140             return self.point(int(idx))
    141         except (ValueError, TypeError):  # Passed variable or variable name.
--> 142             raise ValueError('Can only index with slice or integer')
    143
    144     def __len__(self):

ValueError: Can only index with slice or integer

Versions and main components

  • PyMC3 Version: 3.8
  • Theano Version: 1.0.4
  • Python Version: 3.7.4
  • Operating system: Ubuntu
  • How did you install PyMC3: conda
1 Like
2

Hi,
Have you tried az.from_pymc3(trace)?

3

From the error message, it looks like there is an issue retrieving the pointwise log likelihood values (that is evaluating the logp_elemwise of the model).

Make sure you are using latest ArviZ version (>=0.7.0), using

az.from_pymc3(trace, model=model, log_likelihood=False)

should solve the error. The PyMC3Converter class is used internally by from_pymc3, the result will be the same between the two approaches. The model argument can be omitted if from_pymc3 is called within a model context.

If you were to need log likelihood values (to calculate waic or loo for example) the issue with logp_elemwise should be solved. Common problems when obtaining pointwise log likelihood values are having modified the trace after sampling (adding or removing variables) or sampling only a subset of the latent variables.

4

@OriolAbril A bit late here. I was trying to convert pymc3 trace to inference data with the same code you mentioned above. However, I get the following error:

ValueError: Using the `InferenceData` as a `trace` argument won't work. Please use the `arviz.InferenceData.extend` method to extend the `InferenceData` with groups from another `InferenceData`.
5

@Dawar-812 I used the extend in the GEV example Generalized Extreme Value Distribution — PyMC3 documentation we previously talked about - does this help at all?

6

@ccaprani Thanks for the reply. Unfortunately, it doesn’t work. However, I will try to revisit the problem soon. I intend to do a sensitivity analysis, prior predictive checks would help a lot in that case.

7

@Dawar-812 can you create a new topic with your issue and a minimal example to reproduce the error you encounter? I for example will probably be able to help but I can’t do anything with the error message alone, I need the code that generates the error and the versions for pymc and arviz you use at the very least.

9

@OriolAbril Thanks, here it is