Memory Leakage on Local Machine and AWS

Questions v5
, , ,
1

Hi,

I am having a problem of memory leakage. My input dataset is barely 200Mb.
Shape of dataset is 1.2M x 19.

I have used my local instance to run sampling. It would work fine on a smaller sample of the dataset. But when I plug everything in, it starts consuming more and more memory, until it starts swapping to disk and eventually kills the session.

I tried on heavy AWS Sagemaker compute but it does the same thing over there.

Using PYMC v5. Python 3.9.

Can anyone please help me on this?

@twiecki I could see in another thread this has happened with someone else a few years ago. Is this issue still persistent?

2

Are you using the default sampler?
Are you running on a single machine?
Does it work locally?

3

Hi @twiecki
Thanks for responding. I found out some bottlenecks in the code. Basically there were some multiplications between shapeful and shapeless entities (X,1) * (X,) , this was slowing the code down.
I could resolve the issue.

I was using the default sampler, was getting same results in local as well.

Thanks for checking though :slight_smile:

1 Like