GPU is still experimental as we haven’t yet fixed all the issues. It’s thus possible that there is a lot of data communication during sampling which would slow things down. You could try https://github.com/pymc-devs/pymc3/pull/2345 which might give you better speeds.
NUTS slowing down could mean that initialization failed and once you get to the typical set the mass matrix is actually not well tuned. Here you could try https://github.com/pymc-devs/pymc3/pull/2327 which might do a better job.