Thanks hugely both for those comments! these are super helpful, even just trying to explain and discuss the problem.
About the regression model, it has two reasons to be there. First, it might not work, but I am exactly testing the validity of the regression in this case. So it doesn’t work it is still a result. Second, it is true that I showed 4 picks depending on the distance d, but they might be many more, making the band almost continuous, and they are all related to the same variable (it is a distance), so it is not conceptually right to manage them as unrelated, different categories… but either as a continuous regression, so I am trying to find a spot in the middle.
The model that I tested should be this one:
\begin{align*} \mathrm{\alpha_{0i} } &\sim \mathrm{LogNormal}(0,1) \ , \ i\in \{1,...,N\} \\ \mathrm{\bar{\omega}_0 } &= (\omega_{01},...,\omega_{0N})\sim \mathrm{Dir}(N,\alpha_0)\\ \mathrm{\bar{\omega}_c} &= (\omega_{c1},...,\omega_{cN}) \sim \mathrm{Dir}(N,\omega_0) \ ,\ \ \forall c\in \{1,...,C\} \\ \bar{\nu_c} &= log(\bar{\omega}_c) \\ y\mid c & \sim \mathrm{Normal}(X_c \cdot \bar{\nu_c}, \sigma_y) \end{align*}
It should be hierarchical on \omega_0 to multiple \omega_c, all vectors to normalise with the Dirichlet distribution. I also don’t know if this is correct, really glad to get any feedback on this as well.
+1 to @Armavica . The regression model here confuse me a bit, especially given that
yis essentially a Categorical Random Variable (it is a transformation of one), the parameterpshould have shape(len(y), len(d))which is(85, 4). But your regression output is(85, ), which could not be reshaped into the rightpfor a Categorical. I usually find expressing your problem as a generative model helps, for example, writing a function to simulate fake data withnumpy.random.
This is useful, @junpenglao also because at this point I don’t think the value to interpolate is categorical, but rather just discrete. However, I don’t really know how these types of situations are addressed with probability models.
I might actually try something like
p_d = \mathrm{logit^{-1}}(X_c \cdot \nu_c) \\
w \sim \mathrm{Multinomial}(D,p) \\
y \sim \sum_{d}^{D} w_d \cdot \mathrm{Normal}(\mu_d, \sigma_{yd})
where d\in\{1...D\} is the index for the possible values of y.
Do you think this makes sense in some way? or not completely?
On top of that, the main problem I have at the moment is dimensionality, as you said, guys. I need to understand where the 85x85 comes from first and address that.
Thanks for the huge help and let me know if you have other thoughts
EDIT: I found that the dimensionality issue were coming from a missing .reshape((-1,1)) . This part should be solved