NUTS uses all cores


#1

I am running pymc3 on a machine with a large number of cores (>32) and when sampling using NUTS all of the cores are being utilized. I have set the following environment variables:

MKL_NUM_THREADS=8
OMP_NUM_THREADS=8

but I still observe the behavior in NUTS though Metropolis is well behaved. Are there any other variables I should set to limit the core usage?


#2

Some of the theano_ops will use all the cores, for example in the GP module. I think @bwengals knows a bit more on this.


#3

It is strange because I was alright using NUTS on some of the problems I was working on yesterday but now something seems to have changed.

If there is a way to limit NUTS to only use a specific number of cores that would be very helpful.


#4

The reason it worked before was user error. I was editing a jupyter notebook and some variables were stored that I didn’t recognize.


#5

Do you use MvNormal distribution in your model?


#6

Yes I am.


#7

Yeah from Multidimensional gaussian process

The matrix operations used by Theano here are multithreaded, so running multiple chains simultaneously bogs things down.

I am not sure how to limit it to single thread per chain tho.


#8

Okay, it isn’t an issue at the moment but it would be helpful if that was something we could set in the future.