Reproducibility across machines

Questions
1

Hi all,

I am having trouble getting the sampling traces to be reproducible on different machines. While running the same code multiple times on the same machine gives identical results, running the code on a different machine gives different results. They are very close, but not identical.

They have the same versions of pymc3 (3.6), theano (1.0.4), numpy (1.16.0), and scipy (1.2.0), although have different versions of Python (3.7.2 and 3.6.7). They are both on linux (current Arch, ubuntu 18.04). They do have different compilers installed, but both have empty theano configs.

For testing, I have been running sample as pm.sample(1000, nuts_kwargs={"Emax":700}, cores=4, chains=4, random_seed=[1, 2, 3, 4]).

Could I please check that I can indeed expect the traces to be identical in this scenario? I.e. is having the same model, the same random_seed value, and the same pm.sample parameters sufficient?

Thanks for any help or advice.

2

It is very difficult to get exactly the same traces for different machines. In the test suite, we run a test to check if the step methods (e.g. NUTS, metropolis, SMC, etc) give a trace that “matches” the trace we expect to see. This match is not an absolute equality match (assert all(x == y)) but a more relaxed match (np.allclose) with a tolerance that depends on the floating point precision being used. With that said, across different machines there are some small differences in the traces, and as the sampling progresses, these differences may pile up and lead to very different traces at the end.

3

Great, thank you for the info. For additional context, I am creating an environment on codeocean that I can refer to in a journal article to reproduce an analysis. It sounds like it isn’t as straightforward as using random seeds to get identical results on there and on my local machine.