I’m trying to implement a Gaussian Optimization based on GP regression module in PyMC3. Since I want to extend my model with each new point sampled from the search space, I’m thinking to clear the existing model and re-initiate a new model with a new covariance that has more dimensions ( because for each point we take as a new data-point we have to increase the dimensions of the covariance function).
Now, I’m stuck at clearing the existing model, If I don’t clear the existing model it will lead to a memory leakage due to the each new model created after observing a new data-point. How can I clear the exiting graph with theano (or remove the existing model)?
This is sample code for model initiation :
with pm.Model() as self.model:
l = pm.Gamma("l", alpha=2, beta=1)
nu = pm.HalfCauchy("nu", beta=1)
cov = nu ** 2 * pm.gp.cov.Matern32(X.shape, l)
gp = pm.gp.Marginal(cov_func=cov)
sigma = pm.HalfCauchy("sigma", beta=1)
y_ = self.gp.marginal_likelihood("y", X=X, y=y, noise=sigma)
map_trace = [pm.find_MAP()]
is something like
del model works? I am not sure as well usually what is the best way for this, @colcarroll? Also @bwengals to see if there is another way to do this by changing the GP implementation a bit (as shape is growing it is difficult)
@junpenglao I don’t think
del model is working, I tried that but still the memory usage is increasing with each new model created.
I’ve had this issue as well. I would have to restart notebook/process whenever I wanted to retrain or rerun GP based models because the model’s memory wouldn’t free up.
I’ve noticed that:
- I have only seen this behavior in PyMC3 with GP models
- It happens for all the GP models/implementations (
gp.*, not just e.g.
- The GP implementations in PyMC3 all sit on top of Theano and it’s linear algebra routines, no custom ops or anything in there.
I’m not positive at all, but I think the problem may be with a Theano op that all the GP models use, but isn’t commonly used in other PyMC3 models. Maybe something like
tt.Solve is to blame?
Was a solution to this ever found? I seem to be running into the same issue in pymc3 3.10.0, and
del model doesn’t fix it.