How to model compare on time series

Hello all,

I am trying to identify how to compare models using a non-stationary time series. To add some context, I’m building a hierarchical model where parameters linearly covariate with variables such as time or other physically ones (as an attempt to explain some of the trend pattern), this I call non-stationary model. Then, compare with stationary model.

When model comparing I’ve came across with some concerns regarding the use of future data to test past data (then, the use of LOO). I do not think I have statistical background to replicate the alternative Leave-Future-Out as I haven’t found examples on PyMC3 yet. Is there anything I am missing and LOO/WAIC can be used for time series data if my model built correctly? How to model compare using a non-stationary time series?

I’d appreciate any help and reference indications. Thanks.

You can have a look at this blog post Cross-validation, LOO and WAIC for time series | Statistical Modeling, Causal Inference, and Social Science

Usually non-stationary means whether y has a random walk or not. If yes, you might have to use differencing.


Thank you for pointing this article.

So, we can still use LOO even if we are interested in making predictions. Even though we might get “optimistic estimates of the expected predictive performance”.

Is it correct to assume that, if I am trying to compare models and choose the best one among them, bias would not affect my choice? (as I am subjecting all models to the same bias)

Or am I oversimplifying the issue?