I’m not a stats expert by any means, but I always thought the half-cauchy priors on variance were silly and overkill. Same goes for exponential. Have you ever actually drawn samples from these? They’re nuts.
I’ve been defaulting to Gamma(2, x) for strictly positive variables for a while now, and I haven’t felt the need to look back. It avoids these problems by explicitly ruling out zero. Sometimes when it’s not possible, or when I know ahead of time the variance is going to be close to zero, I’ve had good success switching to modeling the precision instead of the variance. I don’t exactly know what PyMC does under the hood to convert between parameterizations, but the sampler seems to be happier going after big numbers in the tail then flipping them.