Thanks to Bill Engels for an example to how to build a GP model with non-conjugate likelihood!
https://bwengals.github.io/non-conjugate%20GPs.html
I tried to run it with the most recent Theano and PyMC3 installed. Unfortunately I get the nasty FloatingPointError when the model is built and NUTS uses ADVI to initialize.
The complete error message can be found here:
https://pastebin.com/BiM78urj