Hi @bwengals . Thanks for the pointers!
K is 4000 x 4000. Which should be reasonable? It takes about 12 GB of RAM and 4 days to run on my laptop (wish I had access to something better), which is on the edge of me being able to use it for my case. It sure seems its due to repeated inversions of this matrix when sampling as I have tried stronger priors (although not strong gammas-will give that a try). I have also set either eta or lengthscale to fixed values and I get basically no speed ups unless I fix both, which leads me to believe its mostly due to the matrix inversions. But I am fairly new to mcmc so correct me if this logic doesn’t sound right.
I have used inducing points in the marginalsparse method, however they don’t really make sense for my problem as I understand it (and I take your point about integrating over f ). As I want to infer something about each x in K. I do not need all the correlations between x_i's so I could easily make K 80-90% sparse and this could actually help in the inference in my case. What are your thoughts about representing it as a sparse matrix to theano? Are there speed ups here?
Right now I am doing sensitivity analyses by fixing the kernel hyperparameters 
So any speed up that would allow me to sample the posterior of them would be awesome! 
Thanks for taking a look at this!
Brian