Understanding the Predictive Covariance of Gaussian Processes

Questions
1

gp.predict(test_X, point=MAP, diag=True)

I have implemented the rbf regression (squared-exponential kernel) using PyMC3 GP module. I want to estimate the error (uncertainty) of the predictions for each prediction. The predict function is expected to provide mean vector and the covariance matrix (or diagonal of the covariance matrix) of the predictive densities as the outputs.

I observed that the covariance is always in [0, 1].

  • Why can’t it be more than 1?
  • How do we interpret the uncertainty of predictions using the covariance?
2

The covariance can certainly goes above 1 - the pattern you observed is likely model specific.

You need to marginalised the MvNormal to get uncertainty for each predicted output. It might be more straightforward to use sample_ppc for that purpose: http://docs.pymc.io/notebooks/GP-Marginal.html#Using-.conditional