Parallel processing with different distributions causing Theano.gof.compilelock warning

Hi,

I am currently trying to run pymc3 with lammps (MD code), passing lammps potential values (floats) for each run. This works very well within pymc3.Model block, using pymc3.Simulator and pymc3.sample when using a Normal distribution (passing mu and sd). However when I switch out the distribution type to Uniform (passing lower and upper) it doesn’t work. From what I can gather, for Normal you are passing a dict of values around, whereas Uniform is passing Elemwise{add,no_inplace}.0 (Theano object), which can not be read by lammps. To fix this I have use .eval() to return the float value for Elemwise{add,no_inplace}.0. This runs, but keeps throwing up the following messages repeatedly:

INFO (theano.gof.compilelock): Waiting for existing lock by process ‘77206’ (I am process ‘77113’)
INFO (theano.gof.compilelock): To manually release the lock, delete /Users/lmm/.theano/compiledir_Darwin-17.7.0-x86_64-i386-64bit-i386-3.7.3-64/lock_dir

I have also noticed that when running with parallel=True (Default) it doesn’t utilise full capacity of all cores, but combined maxis at 100% (i.e. over 4 cores, it will run something like 70%, 10%, 15%, 5% respectively). When I set parallel=False I no long receive the Theano warnings, and the single core runs at ~100%. It is however exceptionally slow.

I don’t have these issues when using a Normal distribution, which leads me to believe the issue is with passing the Elemwise{add,no_inplace}.0 object instead of the explicit value, or running .eval()?

I have searched through a lot of Theano troubleshooting, adding bits to the .theanorc etc, but nothing prevents the warnings or core capacity issues.

Any suggestions would be most appreciated!

2 Likes