This is a really interesting question and I would love to know more about this idea as well since it relates to similar problems in my work. I have explored this same idea but in a performance context. Ive seen this applied in sports where there is some expected decline with age. The solution I have seen for problems like this is to calculate the delta across years for each individual. Then average the deltas at each time bin (e.g, age) and fit a single curve that can be applied to all. This curve explains the delta w.r.t. the time axis. In theory, this can be used to predict further in time given a starting point. e.g., In your case, you have the measurements and you could simply age out from the most recent observation. My issue with these solutions has always been that there is never a single curve that fits all individuals and that there is some history that needs to be taken into account, such as initial conditions and performance (disease progression in your case) thereafter.
One solution I have tried is to use hierarchical splines or hierarchical HSGPs to fit a curve to each individual. I’ve only had minor success with this since the number of individuals in my data is similar to yours (in the thousands). I actually put together an example of this that is HEAVILY based on example notebooks from the pymc examples (1, 2, 3). The main reason I wrote my own is because in the examples by @bwengals, the length of each series was the same. I wanted to understand if we could apply hierarchical splines/HSGPs if the series were of different length. It just took a little bit of thinking about the indexing, but it did seem to work very well for the toy problem. This is an approach I am actively exploring since it does allow for plateaus/increases.
I’ve thought about approaching this in a few other ways:
-
If the curve is smooth (e.g., progressive decline of illness), then I visualize this in my mind as a basic equation of motion of a projectile. In the simplest case, if we have the initial velocity and angle, then we can calculate the path of the trajectory over time. In your case, one could think of the problem as each individual as having their own initial conditions plus added noise. This of course would make strong assumptions about the smoothness of the decline and may violate what you mentioned about plateaus and improvements. However, this may not be totally unacceptable since estimating plateaus/increases may be difficult w/o intervention. Additionally, estimating those parameters would be straightforward since its like fitting nonlinear regression in a traditional hierarchical format.
-
I’ve explored the idea of fitting these types of curves using something like a hierarchical prophet; however, when I tried this I struggled with sampling/divergences. I would need to think quite a bit more about the components of the model if I were going to go down this route. I do think it is possible to have this model work for this application. If I am thinking about patient decline correctly, I imagine this something like a constant offset over time + some slower Fourier component + faster Fourier component to capture noise. I’m just blindly shooting from the hip here so that may be completely wrong.
-
A more advanced, and likely powerful, way to think about this problem is as an autoregressive problem. I’ve tried to dig into this quite a bit but had a hard time conceptualizing how I would go about solving this. I’ve played around with
pm.ARbut never got too far. In reality, since there are evolving dynamics where each time point is highly informed by the prior observation(s) (+ noise), this seems like the perfect place for a Kalman filter. I am really interested to dig a bit more intopymc_extras.statespacefor something like this. I would recommend you watch the video by @jessegrabowski about his work onpymc_extras.statespace(discourse post). I am currently still working through the classic Time Series Analysis by State Space Methods, so I am not confident in my ability to provide guidance on this approach. If Jesse has any thoughts on this I would love to get his input!
I’m sorry I am not providing a more clear answer on structural time series, but I did want to throw my two cents out there since I have been thinking about basically the same problem. Hopefully we can get a good discussion going on this topic!
– Justin