Here is a simpler example for a 2 parameter model. There is a 1D pipe of length 1 meter, through which water can flow in either direction. The water that flows from the left is always blue. The water that flows from the right is always red. On the boundary between red and blue, the water mixes, resulting in purple water. Hidden state p1 denotes the end position of blue water and start of the purple water. Hidden state p2 denotes the end of purple water and start of red water.
- We know the flowrate direction and approximate value, but there is noise of known distribution.
- We have approximate differential equations which denote how much mixing happens, depending on flowrate and past state. Thus, we have an estimate for hidden states p1 and p2, given their values one timestep ago, and flowrate at this timestep.
- We measure the color of the flow at select locations in the pipe. The locations of the measurement devices are known. The color measurements are noisy, with known distribution. (Normal distribution, all measurement devices are independent).
As you can see, if I turn on the flow in the positive direction (left-to-right) for a long time, all water in the pipe will be blue. During this time, red and purple water will have flown out of the right end of the pipe, resulting in p1 = p2 = 1. Similarly, with the negative flowrate, eventually p1=p2=0. Clearly, we must also have p1 <= p2, because the size of purple water L = p2 - p1 is non-negative.
Now, it is not particularly hard to get rid of the second constraint by reparameterization, we could simply use p1 and L / (1 - p1) as parameters, then both parameters are independent and are within [0, 1], although highly correlated. Yes, the second parameter results in 0/0 when p1 = 0, but we can use some smooth function that reduces to 0.5 when p1 is small without any significant loss of accuracy. The real problem is with converting the interval [0, 1] to a real line, while allowing the state where purple water has been completely pumped out of the system in either direction.
Could you elaborate on how one might move to an unconstrained latent space in my example?