# Trace\_to\_dataframe errors after trace.remove\_values

**URL:** <https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256>\
**Category:** Questions\
**Created:** [June 10, 2020, 5:18pm UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256 "2020-06-10T17:18:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![stst](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/stst/32/1999_2.png) [@stst](https://discourse.pymc.io/u/stst)\
**Post date:** [June 10, 2020, 5:18pm UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/1 "2020-06-10T17:18:14Z")

</div>

Hi PyMC3 community,

I ran into the following issue with PyMC 3.8: `trace_to_dataframe` raises a `KeyError` if called on a trace that had a RV removed via `remove_values`. Here is a MWE:

```auto
import pymc3 as pm
import numpy as np

data = np.random.normal(loc=2,scale=0.5,size=10)

with pm.Model() as m:
    mu = pm.Normal("mu",mu=3,sigma=1)
    sig = pm.InverseGamma("sigma",alpha=1,beta=1)
    pm.Normal("d",mu=mu,sigma=sig,observed=data)

trace = pm.sample(model=m,chains=2,cores=1)

trace.remove_values("sigma")
pm.summary(trace)
df = pm.trace_to_dataframe(trace)
df.head()

```

The error is

```auto
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Untitled-1 in 
     14 trace.remove_values("sigma")
     15 pm.summary(trace)
---> 16 df = pm.trace_to_dataframe(trace)
     17 df.head()

~\miniconda3\envs\dive\lib\site-packages\pymc3\backends\tracetab.py in trace_to_dataframe(trace, chains, varnames, include_transformed)
     36 var_dfs = []
     37 for v in varnames:
---> 38 vals = trace.get_values(v, combine=True, chains=chains)
     39 flat_vals = vals.reshape(vals.shape[0], -1)
     40 var_dfs.append(pd.DataFrame(flat_vals, columns=flat_names[v]))

~\miniconda3\envs\dive\lib\site-packages\pymc3\backends\base.py in get_values(self, varname, burn, thin, combine, chains, squeeze)
    471 try:
    472 results = [self._straces[chain].get_values(varname, burn, thin)
--> 473 for chain in chains]
    474 except TypeError: # Single chain passed.
    475 results = [self._straces[chains].get_values(varname, burn, thin)]

~\miniconda3\envs\dive\lib\site-packages\pymc3\backends\base.py in (.0)
    471 try:
    472 results = [self._straces[chain].get_values(varname, burn, thin)
--> 473 for chain in chains]
    474 except TypeError: # Single chain passed.
    475 results = [self._straces[chains].get_values(varname, burn, thin)]

~\miniconda3\envs\dive\lib\site-packages\pymc3\backends\ndarray.py in get_values(self, varname, burn, thin)
    286 A NumPy array
    287 """
--> 288 return self.samples[varname][burn::thin]
    289 
    290 def _slice(self, idx):

KeyError: 'sigma'

```

Is there a workaround for this? I also see that `trace_to_dataframe` is going to be deprecated and removed ([issue #3907](https://github.com/pymc-devs/pymc3/issues/3907)). What should I use instead to convert a trace to a Pandas dataframe?

Thanks!

---

<div class="post-metadata">

**Author:** ![AlexAndorra](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/alexandorra/32/9142_2.png) [@AlexAndorra](https://discourse.pymc.io/u/AlexAndorra)\
**Post date:** [June 11, 2020, 9:01am UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/2 "2020-06-11T09:01:21Z")

</div>

Hi,  
Yeah I think removing an RV messes up with the trace’s metadata, which would explain the error. I think I remember that @OriolAbril already encountered this case, didn’t you?

As for handling the traces, PyMC’s plots and diagnostics are now handled by [ArviZ](https://arviz-devs.github.io/arviz/), which uses `InferenceData` objects to manipulate traces. This is much more powerful than pandas DataFrames or numpy arrays to handle multidimensional traces, as is very often the case in Bayesian modeling.  
In particular, you can name the dimensions of the variables and use them to slice the trace, without having to bother with shape handling. I would advise [reading the quickstart](https://arviz-devs.github.io/arviz/notebooks/Introduction.html) for a more detailed example.

Hope this helps 🖖

---

<div class="post-metadata">

**Author:** ![OriolAbril](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/oriolabril/32/2497_2.png) [@OriolAbril](https://discourse.pymc.io/u/OriolAbril)\
**Post date:** [June 11, 2020, 7:19pm UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/3 "2020-06-11T19:19:52Z")

</div>

As @AlexAndorra says, in most cases using xarray datasets will fit better the data stored in the trace object which is bound to be multidimensional and not table-like. Moreover, [`arviz.from_pymc3`](https://arviz-devs.github.io/arviz/notebooks/InferenceDataCookbook.html#From-PyMC3) will work even after removing variables from the trace.

I don’t really know how does `trace_to_dataframe` works, but if you really want to use pandas instead of xarray, `xarray.Dataset` objects have a converter function [`to_dataframe`](http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_dataframe.html#xarray.Dataset.to_dataframe). Thus, something like this would produce a dataframe (hopefully similar to the one generated by `trace_to_dataframe`):

```auto
idata = az.from_pymc3(trace, model=m)
df = idata.posterior.to_dataframe()

```

* * *

Having said that, I would argue that most situations do not really require removing variables from the trace. For simple cases, using `var_names` argument in summary and plots would suffice, and for more complicated ones (or for convenience with simpler ones), you can create a subset of the `posterior` group (no extra memory usage) and work with it:

```auto
idata = az.from_pymc3(trace, model=m)
data = idata.posterior[["mu", ...]] # select only desired variables

```

This would have the main advantage of still having `sigma` if it were needed eventually _without having to rerun the calculations_. For example, to calculate pointwise likelihood values (needed for `waic` or `loo`), having all the variables in the original trace is required, as well as all variables are required for `sample_posterior_predictive` (which can be used either for posterior predictive checks or to predict new values based on the model)

---

<div class="post-metadata">

**Author:** ![stst](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/stst/32/1999_2.png) [@stst](https://discourse.pymc.io/u/stst)\
**Post date:** [June 11, 2020, 10:20pm UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/4 "2020-06-11T22:20:54Z")

</div>

Thank you both, this is very helpful!

Should I file an issue regarding `remove_values`?

The only reason I am converting to a pandas dataframe is to generate a CSV file that I could then import into an old MATLAB workflow (which I will port over to Python stepwise). What is the most efficient route to save a pymc3.MultiTrace as a CSV file?

---

<div class="post-metadata">

**Author:** ![AlexAndorra](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/alexandorra/32/9142_2.png) [@AlexAndorra](https://discourse.pymc.io/u/AlexAndorra)\
**Post date:** [June 12, 2020, 7:54am UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/5 "2020-06-12T07:54:05Z")

</div>

Glad this was helpful!  
I’m not sure this is an issue actually: as Oriol said, removing variables from the trace will prevent computing information criteria like WAIC or LOO as well as doing posterior predictive checks. This is a pretty big dent in the principled Bayesian workflow, so I’m not sure making it easier is the way to go.

As for saving to a CSV, I would look at [xarray doc](http://xarray.pydata.org/en/stable/) – as they are based on pandas, there is a chance they have a way to directly save to CSV; or at least you can do to\_pandas then to\_csv, as Oriol said.  
I don’t know if Matlab reads net\_cdf files, but just want to highlight that [ArviZ can save the trace to net\_cdf](https://arviz-devs.github.io/arviz/generated/arviz.to_netcdf.html#arviz.to_netcdf), in case it’s useful 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![stst](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/stst/32/1999_2.png) [@stst](https://discourse.pymc.io/u/stst)\
**Post date:** [June 12, 2020, 7:06pm UTC](https://discourse.pymc.io/t/trace-to-dataframe-errors-after-trace-remove-values/5256/6 "2020-06-12T19:06:13Z")

</div>

Thank you, NetCDF is the solution! One can even skip the `InterferenceData` intermediate.

For future reference, here’s the full workflow:

Python 3.7.6 / PyMC3 3.8 / ArviZ 0.6.1:

```auto
import pymc3 as pm
import numpy as np
import arviz as az

data = np.random.normal(loc=2,scale=0.5,size=10)

with pm.Model() as m:
    mu = pm.Normal("mu",mu=3,sigma=1)
    sig = pm.InverseGamma("sigma",alpha=1,beta=1)
    pm.Normal("d",mu=mu,sigma=sig,observed=data)

trace = pm.sample(model=m,chains=2,cores=1)

az.to_netcdf(trace,"results.nc")

```

MATLAB:

```auto
mu = ncread('results.nc','/posterior/mu');
plot(mu)

```