Setting and justifying priors for a discrete "what went wrong" model when I have no labeled data

Questions v3
1

I’m modeling a data-quality decision as a small discrete belief problem and would
love a sanity check from people who do this properly (I’m a software engineer,
newer to Bayesian methods).

For each scraped product-spec value I keep a belief over 6 states — correct /
unit_error / wrong_product / typo / missing / garbled — start from a
source-reliability prior, update with a likelihood for cheap evidence (e.g. value
out-of-plausible-range), and then choose an action (accept / repair / get more
evidence / flag to a human / reject) by minimizing expected cost (an asymmetric
loss where publishing a wrong value ≫ flagging a good one).

Two things I want to get right:

  1. My priors are reasoned assumptions, not counted from labeled data. What’s the
    least-bad way to set and later calibrate discrete priors like this cheaply?
  2. Is choosing the action by minimum expected loss (with an explicit abstain/flag
    option) the right framing, or am I missing a standard tool here?

Happy to share the full write-up if useful, but mostly want to know where the
reasoning is weak.

2

Hi, @ChiragG2101.

I think you’re going to find the likelihood is going to make a bigger difference. For Bayesian modeling, Don Rubin suggested that you think about what you’d do if you had all the data, build a model for that, then infer the values you don’t have. It’s going to be hard to bootstrap without some grounding, though.

Without any labeled data, you can’t evaluate the system, so I suspect you have some labeled data.

Bayesian decision theory can be done conditionally on the posterior, so nothing needs to go into the model about your decision-making weights or preferences. Then as you suggest, it usually goes by minimizing expected squared loss. The squaring is because squared error is what an average minimizes. That is, if you have values x_1, \ldots, x_N, then

\bar{x} = \textrm{mean}(x_1, \ldots, x_N) = \textrm{arg min}_y \sum_{n=1}^N (x_n - y)^2.

You can look at absolute loss, but you don’t typically look at loss itself directly because it can be symmetric and cancel to zero.