I’m trying to run pymc3 models in multiprocess, but I often lose time on the theano compiler waiting for another process’s lock. Here is an example of the logging message this causes:
INFO (theano.gof.compilelock): Waiting for existing lock by process '21774' (I am process '29874')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/markosestovic/.theano/compiledir_Linux-4.14-el7.centos.x86_64-x86_64-with-centos-7.6.1810-Core-x86_64-3.6.5-64/lock_dir
The second message suggests I can get around this by deleting the lock_dir. But I wanted to check first what possible negative side effects I could cause by doing this.
My understanding is that this is related to theano caching the compiled C code for a model definition. I don’t think I need to reuse compiled code; every new process uses new data (of a different length) so I think in the end it needs to be recompiled (I’m not sure how much of it can be reused, even though the models are all of the same form).
What I dont’ want to happen is that if I do this too fast, the next process might delete the compiled code for the model in the previous process. Is this a possibility?