How do the posterior weights at different layers look like compared to the true weights? Do you see systematic biases? Too flat? Too peaked?
I checked your model more carefully and indeed a fixed noise should be fine (I didn’t realize it was data you generated). Is the posterior for the noise centered around the correct value of 1?
Also, I am not sure I understand your first plots were you demonstrate the noise issues. Could you explain a bit more what you are plotting?