Removing/Resetting existing Gaussian Process Models

Questions
1

I’m trying to implement a Gaussian Optimization based on GP regression module in PyMC3. Since I want to extend my model with each new point sampled from the search space, I’m thinking to clear the existing model and re-initiate a new model with a new covariance that has more dimensions ( because for each point we take as a new data-point we have to increase the dimensions of the covariance function).

Now, I’m stuck at clearing the existing model, If I don’t clear the existing model it will lead to a memory leakage due to the each new model created after observing a new data-point. How can I clear the exiting graph with theano (or remove the existing model)?

Thanks!

This is sample code for model initiation :

with pm.Model() as self.model:
       l = pm.Gamma("l", alpha=2, beta=1)
       nu = pm.HalfCauchy("nu", beta=1)
       cov = nu ** 2 * pm.gp.cov.Matern32(X.shape[1], l)

       gp = pm.gp.Marginal(cov_func=cov)

       sigma = pm.HalfCauchy("sigma", beta=1)
       y_ = self.gp.marginal_likelihood("y", X=X, y=y, noise=sigma)

       map_trace = [pm.find_MAP()]
2

is something like del model works? I am not sure as well usually what is the best way for this, @colcarroll? Also @bwengals to see if there is another way to do this by changing the GP implementation a bit (as shape is growing it is difficult)

1 Like
3

@junpenglao I don’t think del model is working, I tried that but still the memory usage is increasing with each new model created.

4

I’ve had this issue as well. I would have to restart notebook/process whenever I wanted to retrain or rerun GP based models because the model’s memory wouldn’t free up.

I’ve noticed that:

  • I have only seen this behavior in PyMC3 with GP models
  • It happens for all the GP models/implementations (gp.*, not just e.g. gp.Latent).
  • The GP implementations in PyMC3 all sit on top of Theano and it’s linear algebra routines, no custom ops or anything in there.

I’m not positive at all, but I think the problem may be with a Theano op that all the GP models use, but isn’t commonly used in other PyMC3 models. Maybe something like tt.Solve is to blame?

5

Was a solution to this ever found? I seem to be running into the same issue in pymc3 3.10.0, and del model doesn’t fix it.