Hi,
Thanks!
Yes. I know that it doesn’t make sense for small models / datasets but I’ve had to sample big models in the past and being able to compute metrics while training would be have been super useful. I also suspect that as the methods become more widespread people will want it run deep-learning-like super long trainings in which case being able to monitor the model may be useful.
I started writing an inference « machine » in Rust that is an iterator over samples and the flow feels pretty nice. Being a low-level language this adds very little overhead. Maybe it is implemented like this in TFP already?
Something else I think is interesting in terms of big models is minibatch inference. But again, not sure this is something you want to have in an high-level library right now.
I’m also happy to help with anything (code, docs, examples).