Indexing of Posterior Predictions in State Space Model

2

I think you’re getting the shifting because the posterior predictive appends the estimated initial state to the series. So it starts one timestep before the data starts – do you actually need to shift it backwards in that case. But that said: (Wrong and dumb, see below)

  1. If you set a DateTimeIndex on the data you pass tobuild_statespace_graph, the model will automatically store the dates and propagate them to all of the outputs. I wrote all this because reasoning about adding extra initial steps or not adding extra initial steps is maddening, and I don’t wish it on anymore.

  2. Sampling the posterior predictive is non-trivial, so there are a bunch of helper functions there for you. Do you want the conditional posterior or the uncoditional? The smoothed, filtered or predicted? The hidden states or the observed state? Should measurement error be included? etc etc etc.

    In particular, you can use ss_mod.sample_conditional_posterior, which will give you outputs of the kalman filter (predicted, filtered, and smoothed) using the posterior parameter draws in idata. This means each timestep is conditioned on the data up to (predicted), including (filtered), or given everything beyond (smoothed) the data at time t.

So you can look at the outputs like this:

post = ss_mod.sample_conditional_posterior(idata)

fig, ax = plt.subplots(figsize=(14,4), dpi=144)
x_grid = dat.index

mu = post.predicted_posterior_observed.mean(dim=['chain', 'draw']).values
hdi = az.hdi(post.predicted_posterior_observed).predicted_posterior_observed

ax.plot(x_grid[1:], mu[1:])
ax.fill_between(x_grid[1:], *hdi.values.squeeze()[1:].T, alpha=0.25)
dat.log_y.iloc[1:].plot(ax=ax)

Untitled
Untitled1636×503 88.8 KB

I sliced away the first observation because the predicted value was zero for all draws. That might be a bug, I need to check. It should be equal to the initial states, which are not zero.

Also, I had to play around with the priors a bit, especially the initial acceleration (the 2nd value of initial_trend). A value of 10 is saying that time series could suddenly accelerate by as much as 20, which is more than the maximum range of the data. You’ll still learn the correct mean in this case, but it will cause your credible intervals to blow up.

This problem is worse in the one-step predictions than in the kalman filter or smoother, but we don’t really want to look at those outputs, because when there’s no measurement error they just noiselessly encode the observed data. Those are only interesting for looking at the hidden states (e.g. acceleration)

1 Like