Sorry confusing terminology, by confidence I mean I don’t have a clear way of defining for my project when somebody could stop taking data for the experiment as the gaussian process will infer the data accurately enough. The Gaussian process I run is largely within 95% confidence windows and what I would expect.
And yes at the moment I run two different gaussian processes. One looks at predictive points only on the values I have actual experimental data at, and I run the RMSE on this. The second just runs it on 100 uniformly spaced points and predicts values there, this is the situation where I am struggling to decide how to evaluate at which point the model has enough actual data, as I can’t use RMSE.
Although I do like your idea of how often the held-out values fall within the predictive credible intervals. Could you expand on this slightly as I’m not sure I fully understand, but think it would be really useful. Are you suggesting that as I run the subsets of data, 90% of the time the true data value should fall within that 90% confidence interval, and at the point which it no longer does I could define this as the point when the gaussian model does not have enough data to run accurate predictions.
I will also try implementing that last advice in my model as well and seeing the results as it does make a lot of sense, thank you!