GP regularization effect

[luisroque]

Thank you for the detailed answer Martin!

@Martin_Ingram

The first thing I would probably check is whether it might be reasonable to make a Gaussian approximation to your Poisson likelihood. If your counts are large, this might be reasonable (see e.g. Poisson normal approximation error). Then you could marginalise out the GP, at which point MAP estimation for the hyperparameters usually works fine.

I started with this approach, but I had to change it pretty quickly as I was having negative estimations due to the flexibility of the GPs when fitting the data. I can’t really have that.

@Martin_Ingram

By the way, I just noticed your toy data is apparently continuous – it should be discrete for a Poisson! That might also be causing some numerical issues.

This was a copy-paste error, I am indeed always using counts. Thank you for noticing.

@Martin_Ingram

Another option could be to see if you could estimate the hyperparameters with NUTS on some smaller subset of data, and then turn them into a point estimate, e.g. by just using their mean, when fitting/predicting on your full set of time series.

Do you think that this is a better alternative? I would say that it isn’t because I am actually fitting my model to some subset of the data to recover hyperparameters, fitting it again to a different dataset (that includes the subset but that it is nonetheless significantly different) to recover parameters, and then conditioning on new data. It seems a bit tricky.

@Martin_Ingram

You could also try a sparse approximation to the GPs: Sparse Approximations — PyMC3 3.10.0 documentation . The idea there would roughly be that you approximate the GPs with 300+ points with ones with say 100, which can speed things up.

I also did some digging here, but there is no current solution to do sparse approximations with latent GPs.

@Martin_Ingram

You could do things with variational inference too, there are special techniques developed for GPs (e.g. here) that can work well, but you would likely have to do this outside pymc3.

I am gonna check this reference.

Thank you!