I can’t get it to work under any circumstances. The behavior is always the same: it allocates approximately 100MB of additional memory per second, and allocated memory keeps growing until it exceeds the machine’s physical capacity, at which point the process degrades completely due to memory swapping. I have tried 61GB, 122GB and 244GB machines; all produce the same outcome except that the larger memory instances last for a longer time until they start to thrash.
It is dying during the initialization process using ADVI (v3.1) or jitter+adapt_diag (v3.2) – usually about 20% finished depending on machine specs.
Ubuntu 16.04, Python 3.5.2, Theano 0.9.0
I have tried all combinations of the following:
- PyMC3 v3.2 and v3.1
- Default backend
- HDF5 backend
I would love to use PyMC3 for a computing project, but can’t proceed with any confidence unless I can find a way to complete runs with a reasonably available quantity of memory.
I would appreciate any insight you might have. Is there a known good configuration available within AWS (i.e. a combination of operating system, Python version, PyMC3 version, etc)?
Thanks again for your assistance!