How many threads is it using? You can at least sanity check it with htop or the system monitor or so. If there are more worker threads in total than you have cores something is doing something stupid…
Depending on the blas you are using, you might also have to set for instance OPENBLAS_NUM_THREADS. Which blas recognizes which environment variable can be a bit messy…