Copy/reinitialize a model from an existing model?

Questions
1

Firstly, I can only guess that what I’m about to ask about would require some way to copy the underlying Theano graph for the source model to a new graph for the other model. I have no idea if that is easy or possible in Theano.

So the actual question: I have a source model that is being updated with “actual” observed variables (from “actual” data), then what I want to do is copy and/or re-initialize another model from the source model (and whatever has been added to it since the last copy/re-init) … is this possible?

In extremely rough terms, envisioning something like this:

  • source model starts with vars: a,b,c,d
  • copy/re-init other model to source, now other has vars: a,b,c,d
  • do some stuff with other model, it now has vars: a,b,c,d,x1,x2,x3
  • then do some stuff with source model, it now has vars: a,b,c,d,e
  • copy/re-init outher model to source, and now other has vars: a,b,c,d,e
  • do some stuff with other model, it now has vars: a,b,c,d,e,y5,y6,y7 (but NO x1,x2,x3)
  • then do some stuff with source model, it now has vars: a,b,c,d,e,f

And so on …

Can this be done without (re-)building each model from scratch, at each step???

2

Yes, you can use theano.clone to replace input to a, b, c, d, and then take it from there. While it is possible, I would rebuild each model at each step, cause correctly doing it using theano.clone is quite complicated.

3

OK @junpenglao thanks for the response!

My source model I can just call the context and add variables to it without re-building, but for the other model I will just have to build from scratch each time. Would be “nice” in the future if pymc3 have some kind of high level interface to do something like what I desire. Something like numpy array copying: otherModel = sourceModel.copy() or .clone(), which literally just like a shorthand for building a new model with exactly the same vars/data as sourceModel.

4

Graph editing is generally discouraged in Tensorflow, so I doubt that it would ever be implemented…
In PyMC3 land, you can copy replace the node in a graph using theano.clone, but there is always a concept of a “base” graph like - not sure what would be the best way to copy part of the graph. @ferrine?

5

OK @junpenglao, thanks again. Even if it was never editing the graph … even if there was a .copy() or .clone() method that just straight up copied/cloned the entire graph of a model into a separate copy of the graph for a new model, that would be helpful.

Now:
model A has vars a,b,c
B = A
model B now has vars a,b,c
but if I add var “d” to model B, it also gets added to model A … B is really just a pointer to the one model A

Ideal:
B = A.copy() or A.clone(), now there are two separate graphs/models with exactly the same structure.

6

Maybe you can try something like a copy.deepcopy?

7

If you just want to copy the model, you can try to use copy.deepcopy(model)

8

LOL, we synced :wink:

1 Like
9

Hi! Theano clone works on graph nodes. It can’t perform inplace updates you need (I hope I’ve understood the context). That is why the only option is to recreate the graph from scratch. In your quite complicated workflow I’d create a custom model class / factory function that can conditionally change the specification you desire. This is more elegant and consistent rather than inplace updates or custom manipulations. Memory cost would be the same and it is low (in case you do not create large size constants and share heavy data)

1 Like
10

Thanks for all the replies everyone! I think it turns out that copy.deepcopy(model) is exactly what I need to use in my application! While this obviously wouldn’t be as efficient as what I was thinking about in the original post, I think it will work just fine for me as I’m using (relatively) small models.