Nope. That answer is terrible in so many ways I’m going to need a list.
- It said it would produce a hierarchical model, but didn’t.
- I asked it to generate simulated data to fit, but it only generates simulated parameters—it uses a variable
simulated_datathat doesn’t exist in the model. - It did not solve either of the non-identifiability problems, though it mentioned how to solve them in its initial description.
- It did not generate the code I asked to compare the fit to the simulated values.
- No extra credit—that explanation of SBC doesn’t say how to do it.
On the plus side, it did code the likelihood correctly even if the simulated data doesn’t exist.
I tried the same prompt in ChatGPT o4-mini-high (they seriously need some rebranding) and the result was equally terrible—it got the manual non-centering backwards. And it also failed to write the SBC code for me or even simple evaluation code. See PyMC example chat.
When I asked the same model in a new chat (don’t know what to make of that now that history is always on) to try it for Stan with a Python driver, it got the non-centering right, the simulation right, and the SBC right, but still couldn’t figure out how to identify the model. Which I guess puts it right about where our mid-level users are at—this kind of how-to-identify a model comes up all the time, which is why I chose it as an example. See Stan example chat.