Can't load trace in the simplest case

Questions v3
1

Hi,

I am completely unable to fit a model using an old trace. I have this code (adapted from one of the notebooks):

#!/usr/bin/env python3
import arviz as az
import numpy as np
import pandas as pd
import pymc3 as pm

from pymc3 import HalfCauchy, Model, Normal, glm, plot_posterior_predictive_glm, sample
from pymc3.backends import NDArray

print(f"Running on PyMC3 v{pm.__version__}")

RANDOM_SEED = 8927
rng = np.random.default_rng(RANDOM_SEED)

size = 200
true_intercept = 1
true_slope = 2

x = np.linspace(0, 1, size)
# y = a + b*x
true_regression_line = true_intercept + true_slope * x
# add noise
y = true_regression_line + rng.normal(scale=0.5, size=size)

data = pd.DataFrame(dict(x=x, y=y))

with Model() as model:  # model specifications in PyMC3 are wrapped in a with-statement
    # Define priors
    sigma = HalfCauchy("sigma", beta=10, testval=1.0)
    intercept = Normal("Intercept", 0, sigma=20)
    x_coeff = Normal("x", 0, sigma=20)

    # Define likelihood
    likelihood = Normal("y", mu=intercept + x_coeff * x, sigma=sigma, observed=y)


old = az.from_netcdf("trace.nc")
with model:


    step = pm.Metropolis()
    # Inference!
    # draw 3000 posterior samples using NUTS sampling
    trace = sample(
        3000,
        chains=1,
        step=step,
        return_inferencedata=True,
        trace=old.posterior
    )
trace.to_netcdf("trace.nc")

Running this results in an error: AttributeError: 'str' object has no attribute 'name'

Backtrace 
Running on PyMC3 v3.11.4
Sequential sampling (1 chains in 1 job)
CompoundStep
>Metropolis: [x]
>Metropolis: [Intercept]
>Metropolis: [sigma]
Traceback (most recent call last):------------------------------------------------------------------| 0.00% [0/4000 00:00<00:00 Sampling chain 0, 0 divergences]
  File "/home/jas/surfd_sync/Documents/experiments/exp5/tmp/./pymc_test.py", line 44, in <module>
    trace = sample(
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/sampling.py", line 597, in sample
    trace = _sample_many(**sample_args)
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/sampling.py", line 713, in _sample_many
    trace = _sample(
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/sampling.py", line 855, in _sample
    for it, (strace, diverging) in enumerate(sampling):
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/fastprogress/fastprogress.py", line 47, in __iter__
    raise e
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/fastprogress/fastprogress.py", line 41, in __iter__
    for i,o in enumerate(self.gen):
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/sampling.py", line 978, in _iter_sample
    strace = _choose_backend(trace, chain, model=model)
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/sampling.py", line 1381, in _choose_backend
    return NDArray(vars=trace, **kwds)
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/backends/ndarray.py", line 219, in __init__
    super().__init__(name, model, vars, test_point)
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/backends/base.py", line 66, in __init__
    self.varnames = [var.name for var in vars]
  File "/nix/store/9ijdjmpkbi6gcxmh3ab95zfczkfhxc0j-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/backends/base.py", line 66, in <listcomp>
    self.varnames = [var.name for var in vars]
AttributeError: 'str' object has no attribute 'name'

I tried also supplying a list to trace. E.g. trace=[old.posterior["x"], old.posterior["Intercept"], old.posterior["sigma"]], which gives a different error:

TypeError: Outputs must be theano Variable or Out instances. 
Received <xarray.DataArray 'x' (chain: 1, draw: 3000)>

or which was suggested here using: trace=old.posterior.stack(sample =("chain", "draw")). This results in the same error as using trace=old.posterior.

I imagine there is a simple thing I’m missing but I can’t figure out what is is.
I am on pymc 3.11.4, python 3.9.10, and arviz 0.11.4

Update: using pm.save_trace(..) and pm.load_trace(...) works. For anyone searching for this:

with model:

    old = pm.load_trace(directory="traces")
    step = pm.Metropolis()
    trace = sample(
        3000,
        chains=1,
        step=step,
        trace=old
    )
    pm.save_trace(trace, directory="traces", overwrite=True)
2

You can’t combine save_trace and to_netcdf. InferenceData arguments also can’t be used as trace argument. I am not sure what happened when trace was provided, maybe @michaelosthege knows. However, I do know that the mass matrix is not available if multiple chains are samples, so it is impossible it “starts sampling from where the old trace finished” (unless using a single chain).

You can use arviz.concat — ArviZ dev documentation to combine multiple posterior fits on the same model, either concating on the draw dimension (or probably recommended in this case unless you do have the mass matrix) on the chain dimension. That is, keep your current results, rerun pm.sample on the same model and then use az.concat

3

Thank you very much for your reply.

I wasn’t doing that. Sorry, maybe the commented code was misleading. When using arviz, I would first run the code using to_netcdf (with return_inference_data=True) and then use from_netcdf to read the inference data.

I thought old.posterior is an xarray? Do you mean that only MultiTrace can be used as a trace argument to start sampling where the process finished?

That’s exactly my case! I want to use one chain, and start sampling from where I’ve finished. I just can’t figure out how to do it with arviz.

Thanks, that’s really helpful but from what I understand arviz.concat returns an InferenceData object. This still makes it impossible to start sampling from where the old trace finished, right? Do you know if this in general impossible with arviz? This is crucial to me, I am fitting a very slow model on a cluster. The model can fail for many reasons and I need to be able to save the sampling “state”.

Update
Ok, I see there exists a solution for a similar problem for pymc4. I’m not sure yet if this would work with pymc3.

4

Again, I can’t confirm as I don’t know the internals of the trace argument, but I would bet what is happening is not “starting from where the provided trace finished”, it is probably only using the last draw from the provided trace as starting point and nothing else, tuning is probably still needed for example. “starting from where the provided trace finished” is quite convoluted, as mentioned in the same thread you linked.

1 Like
5

For future reference. If you are trying to “restart” a metropolis process i.e. start with a proposal from an old trace (but also keep in mind what was said above, this is not really a restart of the exact same process), and it doesn’t work, try omitting the trace argument. Instead provide start explicitly. E.g. when using sample:

trace = pm.sample(20, start = <your_old_trace>.point(-1),
            step=pm.Metropolis(), chains=1, cores=1,
            discard_tuned_samples=False)

When using iter_sample:

for trace in pm.iter_sample(draws=100,
    chain=0,
    start=<your_old_trace>.point(-1),
    step=pm.Metropolis(),
    tune=0): ...

Start being <your_old_trace>.point(-1) is supposed to be the default behavior when the trace is provided, but this wouldn’t work for me on pymc3 3.11.4 and 3.11.5.
See also this stackoverflow answer: (click)
Of course, <your_old_trace> has to be a MultiTrace object.

1 Like