# Trying to impute missing categorical data

**URL:** <https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093>\
**Category:** v5\
**Created:** [January 1, 2023, 10:36pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093 "2023-01-01T22:36:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![wnorcbrown](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/wnorcbrown/32/5913_2.png) [@wnorcbrown](https://discourse.pymc.io/u/wnorcbrown)\
**Post date:** [January 1, 2023, 10:36pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/1 "2023-01-01T22:36:43Z")

</div>

I’m trying to use a pymc model to impute missing categorical values in my dataset.

When I set the observed data in a Categorical distribution to a masked array:

```auto
data = np.ma.masked_equal([1, 1, 0, 0, 2, -1, -1], -1)
with pm.Model():
    idx = pm.Categorical(f"{var}_idx", p=[0.1, 0.2, 0.7], observed=data)
    print(idx.dtype)

```

this prints ‘float64’. So I can’t use the output as an index in my graph.

If the observed data is not a masked array:

```auto
data = [1, 1, 0, 0, 2, 0, 0]
with pm.Model():
    idx = pm.Categorical(f"{var}_idx", p=[0.1, 0.2, 0.7], observed=data)
    print(idx.dtype)

```

then this prints ‘int64’, as expected.

Any ideas or workarounds? Looks like it might be a bug to be honest.

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [January 2, 2023, 8:34am UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/2 "2023-01-02T08:34:06Z")

</div>

What version of PyMC are you using?

---

<div class="post-metadata">

**Author:** ![wnorcbrown](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/wnorcbrown/32/5913_2.png) [@wnorcbrown](https://discourse.pymc.io/u/wnorcbrown)\
**Post date:** [January 2, 2023, 9:53am UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/3 "2023-01-02T09:53:46Z")

</div>

pm. **version**

‘4.3.0’

and I get the same issue in 4.4.0 and 5.0.1

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [January 2, 2023, 5:49pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/4 "2023-01-02T17:49:08Z")

</div>

Looks like a bug. As a short solution you can cast the result variable

```python
idx = idx.astype("float64")

```

Do you mind opening an issue in the GitHub repo?

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [January 2, 2023, 6:26pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/5 "2023-01-02T18:26:10Z")

</div>

I believe this is a bug related to how the interpolated values are re-combined with the observed values, and opened an issue [here](https://github.com/pymc-devs/pymc/issues/6424).

In the meantime, you can work around this using `at.cast` to manually convert the datatype of `idx`, as in:

```python
data = np.ma.masked_equal([1, 1, 0, 0, 2, -1, -1], -1)
with pm.Model():
    idx = pm.Categorical(f"{var}_idx", p=[0.1, 0.2, 0.7], observed=data)
    idx = at.cast(idx, 'int64')

```

---

<div class="post-metadata">

**Author:** ![wnorcbrown](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/wnorcbrown/32/5913_2.png) [@wnorcbrown](https://discourse.pymc.io/u/wnorcbrown)\
**Post date:** [January 2, 2023, 8:59pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/6 "2023-01-02T20:59:12Z")

</div>

This work around doesn’t seem to work for me.

When I cast to int64, the sampled values aren’t in the range of my categorical variable and hence I get index errors:

```auto
data = np.ma.masked_equal([1, 1, 0, 0, 2, -1, -1], -1)
with pm.Model():
    idx = pm.Categorical("idx", p=[0.1, 0.2, 0.7], observed=data)
    idx = at.cast(idx, 'int64')
    num = pm.Normal("num", 0, 1, size=3)[idx]
    pm.sample()

```

Output:

```auto
    rval = inputs[0]. __getitem__ (tuple(inputs[1:]))
IndexError: index 5 is out of bounds for axis 0 with size 3

```

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [January 2, 2023, 9:04pm UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/7 "2023-01-02T21:04:41Z")

</div>

I ran into this as well when trying to track down the first bug, I was hoping it was just something wrong with my system. I think there’s a glitch that sets the shape of the observed data as the number of classes in the categorical (rather than inferring it from the length of `p`).

Could you test that:

1. There’s no error if the length observed data (still with missing values) is exactly equal to the length of p, and;
2. If (1) works successfully, that if you sample from data with fewer observations than the number of classes, the largest class you get in your samples is the length of the data, not the length of p?

If I’m right I think I can track the bug down pretty quick.

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [January 3, 2023, 12:24am UTC](https://discourse.pymc.io/t/trying-to-impute-missing-categorical-data/11093/8 "2023-01-03T00:24:18Z")

</div>

I opened a separate issue for this bug here:

> <https://github.com/pymc-devs/pymc/issues/6427>
>
> \### Describe the issue:
> 
> Originally raised on discourse \[here\](https://discour…se.pymc.io/t/trying-to-impute-missing-categorical-data/11093/7). 
> 
> It appears that when \`pm.Categorical\` has \`observed\` data with missing values, the variable is re-instantiated inside \`model.make\_obs\_var\` with the wrong number of categories. This example shows that the new number of categories is indeed controlled by the shape of the data:
> 
> \`\`\`python
> import pymc as pm
> import numpy as np
> 
> \# No error
> data = np.ma.masked\_equal(\[1, -1, -1\], -1)
> with pm.Model():
> idx = pm.Categorical(f"hi\_idx", p=\[0.1, 0.2, 0.7\], observed=data)
> pm.draw(idx, 100).max()
> \>\>\>Out: 2.0
> 
> data = np.ma.masked\_equal(\[1, -1\], -1)
> with pm.Model():
> idx = pm.Categorical(f"hi\_idx", p=\[0.1, 0.2, 0.7\], observed=data)
> pm.draw(idx, 100).max()
> \>\>\> Out: 1.0
> \`\`\`
> 
> If the data are longer than the number of classes, the code will error out, as shown below:
> 
> \### Reproduceable code example:
> 
> \`\`\`python
> import pymc as pm
> import numpy as np
> 
> data = np.ma.masked\_equal(\[1, 1, 0, 0, 2, -1, -1\], -1)
> with pm.Model():
> idx = pm.Categorical(f"hi\_idx", p=\[0.1, 0.2, 0.7\], observed=data)
> pm.draw(idx, 100).max()
> \`\`\`
> 
> 
> \### Error message:
> \<details\>
> \<p\>
> 
> \`\`\`shell
> 
> IndexError: index 5 is out of bounds for axis 0 with size 3
> Apply node that caused the error: AdvancedSubtensor1(TensorConstant{\[0.1 0.2 0.7\]}, TensorConstant{\[5 6\]})
> Toposort index: 1
> Inputs types: \[TensorType(float64, (3,)), TensorType(uint8, (2,))\]
> Inputs shapes: \[(3,), (2,)\]
> Inputs strides: \[(8,), (1,)\]
> Inputs values: \[array(\[0.1, 0.2, 0.7\]), array(\[5, 6\], dtype=uint8)\]
> Outputs clients: \[\[categorical\_rv{0, (1,), int64, True}(RandomGeneratorSharedVariable(\<Generator(PCG64) at 0x12AD72820\>), TensorConstant{(1,) of 2}, TensorConstant{4}, AdvancedSubtensor1.0)\]\]
> 
> Backtrace when the node is created (use PyTensor flag traceback\_\_limit=N to make it longer):
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3194, in run\_cell\_async
> has\_raised = await self.run\_ast\_nodes(code\_ast.body, cell\_name,
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3373, in run\_ast\_nodes
> if await self.run\_code(code, result, async\_=asy):
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3433, in run\_code
> exec(code\_obj, self.user\_global\_ns, self.user\_ns)
> File "/var/folders/7b/rzxy96cj0w751\_6td3g2yss00000gn/T/ipykernel\_96907/3148429013.py", line 3, in \<module\>
> idx = pm.Categorical(f"hi\_idx", p=\[0.1, 0.2, 0.7\], observed=data)
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/pymc/distributions/distribution.py", line 457, in \_\_new\_\_
> return super().\_\_new\_\_(cls, name, \*args, \*\*kwargs)
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/pymc/distributions/distribution.py", line 310, in \_\_new\_\_
> rv\_out = model.register\_rv(
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/pymc/model.py", line 1348, in register\_rv
> rv\_var = self.make\_obs\_var(rv\_var, observed, dims, transform)
> File "/Users/jessegrabowski/mambaforge/envs/econ/lib/python3.9/site-packages/pymc/model.py", line 1425, in make\_obs\_var
> (missing\_rv\_var,) = local\_subtensor\_rv\_lift.transform(fgraph, fgraph.outputs\[0\].owner)
> 
> HINT: Use the PyTensor flag \`exception\_verbosity=high\` for a debug print-out and storage map footprint of this Apply node.
> 
> \`\`\`
> \</p\>
> \</details\>
> 
> \### PyMC version information:
> 
> \<details\>
> PyMC version: 5.0.1
> Pytensor version: 2.8.11
> \</details\>
> 
> \### Context for the issue:
> 
> \_No response\_