Your intuition is right. If you want to learn something about x you need to model some structure behind it. @junpenglao had a nice talk on this topic you might find useful as a starting point: Partial Missing Multivariate Observation and What to Do With Them by Junpeng Lao