What are the negative side effects of releasing theano compiler lock by deleting the lock_dir?

Questions
1

I’m trying to run pymc3 models in multiprocess, but I often lose time on the theano compiler waiting for another process’s lock. Here is an example of the logging message this causes:

INFO (theano.gof.compilelock): Waiting for existing lock by process '21774' (I am process '29874')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/markosestovic/.theano/compiledir_Linux-4.14-el7.centos.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.5-64/lock_dir

The second message suggests I can get around this by deleting the lock_dir. But I wanted to check first what possible negative side effects I could cause by doing this.

My understanding is that this is related to theano caching the compiled C code for a model definition. I don’t think I need to reuse compiled code; every new process uses new data (of a different length) so I think in the end it needs to be recompiled (I’m not sure how much of it can be reused, even though the models are all of the same form).

What I dont’ want to happen is that if I do this too fast, the next process might delete the compiled code for the model in the previous process. Is this a possibility?

2

There’s none which is serious from my experience.
At most, it might kill the execution of the model (assuming you do it while running).

3

Any idea which cases it would kill the execution of the model? That would ruin the point a bit; the whole idea is that I would have several processes trying to run samplers/MAP at the same time, and I don’t want them waiting in a queue to compile their models.

Also, is there way to “catch” when this warning pops up? Or should I just check if the lock_dir exists and delete it blindly?

It seems like a weird mechanic, why can’t theano compile multiple models at the same time?

4

Did you find a salution I want theano to compile multiple models at the same time