I think this is a great idea, but I agree. Those diagnostics are plots because they can’t be boiled down to a single statistic. A lot of them aren’t necessarily just binary bad / good indicators too, but they get at what the specific problem might be.
You could try a two-stage approach. Automatically track the top-line metrics, like divergences, ESS, rhats, and then send a summary or alert when model is out of bounds. Then I’d also be sure to save trace plots and other important graphical indicators associated with each run too. Additionally it’s also nice to have some plots that show something about the incoming data, because often the model doesn’t fit right because the data was weird. So while you won’t look at these most of the time, you’ll have them available to diagnose a model with divergences / bad ess / bad rhats.