I have two different prediction models (say one for sales in Walmart and another for sales in JC Penney) , and from each model I have the 5000 samples from the posterior distribution of the percentage error in prediction. I can therefore compute mean percentage error, and the standard deviation of the percentage error. I want to compare whether one model is better at prediction that the other.
The procedure I have in mind is to randomly draw 5000 times a percentage error value from each posterior, compute the difference and then a) plot the differences and b) compute the HPD/HDI and check if 0 lies within the 95% HDI/HPD. If it does include 0, then the two models have similar percentage errors. If not the two are different.
Does that make sense?
Any references would be most welcome.