I worked out how to do this. Here’s a gist that shows an implementation for a simple case: https://gist.github.com/ckrapu/8be4da91a70763ee62f889c6cd98f700
The basic idea is that instead of manipulating the Theano symbolic variables as they come into the step method, work instead with the point object which contains their numerical values.