Hi,
I’m using this model to do some predictions:
def save_model():
with pm.Model() as model:
l = pm.Gamma("l", alpha=2, beta=1)
offset = pm.Gamma("offset", alpha=2, beta=1)
nu = pm.HalfCauchy("nu", beta=1)
cov = nu ** 2 * pm.gp.cov.Polynomial(X.shape[1], l, 2, offset)
gp = pm.gp.Marginal(cov_func=cov)
sigma = pm.HalfCauchy("sigma", beta=1)
y_ = gp.marginal_likelihood("y", X=X, y=Y, noise=sigma)
map = [pm.find_MAP()]
X_New_shared = theano.shared(X)
f_pred = self.gp.conditional("f_pred", X_New_shared, shape=X_New_shared.get_value().shape[0])
I have saved this model with pickle using
with open('save/saved_model.p', 'wb') as buff:
pickle.dump({'model': self.model, 'trace': self.map, 'X_New_shared': X_New_shared, 'f_pred': f_pred, 'scaler': globalScaler, 'encoder': globalEncoder, 'gp': self.gp}, buff)
I’m trying to use this model to do predictions for API calls.
My current approach is to load this model (I know I’m pickling a lot of unnecessary stuff here):
with open(const.model_path, 'rb') as buff:
data = pickle.load(buff)
return {
"model": data['model'],
"trace": data['trace'],
"x_shared": data['X_New_shared'],
"f_pred": data['f_pred'],
"scaler": data['scaler'],
"encoder": data['encoder'],
"gp": data['gp'],
}
and then do the predictions with:
def predict_gp(self):
with self.poly_model:
mu, var = self.gp.predict(Xnew=self.x_shared, point=self.trace[0], diag=True)
return mu
So basically I’m changing x_shared to get new predictions when calling predict_gp()
All this works fine, but the issue is;
[1] Every time I run the predict_gp() method, memory usage increase by about 15MB, and it’s not freed up after the prediction. So as much as I call this method, it keeps increasing until the system runs out of memory.
[2] I find its impossible to call this predict_gp() method concurrently, since it throws errors. I’m currently able to get one prediction after the other. I reckon this is probably because changing shared variable in the middle of the prediction somehow alters the theano graph…
I’m looking at how I can solve this memory issue at the moment. I read several threads relating to similar issues, although not exact: Excessive memory usage in PyMC3? (Solved - AWS Linux platform issue. Works on AWS Windows)
I tried using multiprocessing, and it gets rid of memory issue, but then since it compiles theano every time, each prediction takes about 12 seconds, where as it took around 600ms earlier.
Any thoughts on this?
Thanks