In general, we tend to work with samples even when the posterior has closed form solution. It’s just easier to ask about moments or summaries of interest when you have histograms.
In practice, PyMC is built almost exclusively around mcmc sampling, so it will use that even when not needed. This may be wasteful in some cases but is the most general solution. Your model may have a marginal GP together with other likelihoods and priors / hyperpriors that also need sampling and which wouldn’t necessarily have a known closed form solution. MCMC will handle all these cases naturally.