How to troubleshoot sampling error with custom, not differentiable likelihood?

Perhaps there’s a misunderstanding about what the Bayesian approach is? You’re not optimizing an objective function, you’re generating samples of a fixed posterior distribution. There’s no “improvement” going on as you run the MCMC algorithm, in the typical machine learning sense. It’s mapping out a complex probability distribution that is fixed as soon as you write down the model and provide the data.

This last point is important: you must show the model fixed data. The IoU is not a fixed quantity; it is a function of the current samples. I don’t even think your model as written above will run in PyMC v4, because PyMC now checks that data given to a likelihood function doesn’t link to any random variable nodes in the computation graph. I hope you can see why this would problematic: you have an infinite regression problem. The likelihood depends on samples which depend on the likelihood which depend on samples… If you solved the problem of bad initial values, I suspect this “moving target” problem is related to your current woes.

This is my point when I say you should be modeling your data directly, instead of a model evaluation metric. What does the IOU capture that you think is essential? If it’s spatial dependence, you can run 2D convolution over the temperature, or use a multivariate normal some covariance function (or just model the entire covariance matrix). If it’s something else, reflect on what that something else is and explicitly include it in the latent probability parameter of the Bernoulli distribution.

Or, if you’re really convinced that you must optimize IoU, drop the Bayesian thing and just chuck it into a neural net with an IoU loss function. But I think that reasoning though what you believe IoU is capturing and trying to build that directly into the data generating process is a worthwhile exercise.

2 Likes