Installing nutpie in Docker with pymc5

Questions v5
1

Hello together,

I’ve built a model (successfully) using pymc and for this model, I want to use the nutpie sampler (mainly due to performance reasons, as my model runs in 10 minutes using nutpie compared to multiple hours using the default pymc sampler).

My challenge:
I am currently in the process to bring my model into production - meaning I want to schedule it via Airflow which in our setup requires using a Docker container.

I managed to install nutpie in Docker (having quite some headache) using the following Dockerfile:

FROM python:3.11.2-slim-buster

# - - - - - - UNIX-LEVEL INSTALLTIONS - - - - - - - - 
USER root
RUN apt update && apt upgrade -y \
  && apt -y install --no-install-recommends -qq \
     curl ca-certificates make g++ git vim-tiny openssh-client \
     unzip tzdata htop \
  && apt clean && rm -rf /var/lib/apt/lists/*

RUN apt-get update && \
    apt-get -y install gcc mono-mcs && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && apt-get install -y \
    build-essential \
    libedit-dev  \
    llvm
# - - - - - - - - - - - - - - - - - - - - - - - - - -


# - - - - - - INSTALL PACKAGES - - - - - - - - - - - 
RUN pip install --upgrade pip
RUN pip install "pymc>=5"
RUN pip install "google-cloud-storage" \
                "google-cloud-bigquery" \
                "arviz" \
                "matplotlib" \
                "numpy" \
                "pandas" \
                "seaborn" \
                "rich" \
                "pandas_gbq" 
# - - - - - - - - - - - - - - - - - - - - - - - - - -


# - - - - - - IMPORT FILES AND GRANT ACCESS - - - - - 
COPY . /app
WORKDIR /app
RUN useradd -ms /bin/bash admin
RUN chown -R admin:admin /app
RUN chmod 755 /app
USER admin
# - - - - - - - - - - - - - - - - - - - - - - - - - -



# - - - - - - INSTALL NUPIE SAMPLER - - - - - - - - -
RUN curl https://sh.rustup.rs  -sSf | bash -s -- -y
# RUN apt -y install gcc
ENV PATH="$PATH:/home/admin/.cargo/bin"
#llvm, gfortran
RUN pip install 'nutpie[pymc]'
# - - - - - - - - - - - - - - - - - - - - - - - - - -


CMD ["python", "run_model.py"]

Still when I run a model, I get an error that refers to missing dependencies - but in the web I just can’t find which dependencies are missing:

File "/app/test_nutpie_in_docker.py", line 11, in <module>
    compiled_model = nutpie.compile_pymc_model(model)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/.local/lib/python3.11/site-packages/nutpie/__init__.py", line 7, in compile_pymc_model
    raise ValueError("Missing dependencies for pymc. `import nutpie.compile_pymc` to see error.")
ValueError: Missing dependencies for pymc. `import nutpie.compile_pymc` to see error.

Can someone please help here?

Thanks a lot in advance

2

Can you call a script that does import nutpie.compile_pymc? As the error message says, that should give you more useful information

1 Like
3

Thanks a lot for your hint @ricardoV94

Indeed,this helped - the only missing dependency was a package called numba (installable via pip)

So for anyone who wants to dockerize a pymc-based module using the nutpie sampler, here a working Dockerfile:

FROM python:3.11.2-slim-buster

# - - - - - - UNIX-LEVEL INSTALLTIONS - - - - - - - - 
USER root
RUN apt update && apt upgrade -y \
  && apt -y install --no-install-recommends -qq \
     curl ca-certificates make g++ git vim-tiny openssh-client \
     unzip tzdata htop \
  && apt clean && rm -rf /var/lib/apt/lists/*

RUN apt-get update && \
    apt-get -y install gcc mono-mcs && \
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && apt-get install -y \
    build-essential \
    libedit-dev  \
    llvm
# - - - - - - - - - - - - - - - - - - - - - - - - - -


# - - - - - - INSTALL PACKAGES - - - - - - - - - - - 
RUN pip install --upgrade pip
RUN pip install "pymc>=5"
RUN pip install "google-cloud-storage" \
                "google-cloud-bigquery" \
                "arviz" \
                "matplotlib" \
                "numpy" \
                "pandas" \
                "seaborn" \
                "rich" \
                "pandas_gbq" \
                "numba"
# - - - - - - - - - - - - - - - - - - - - - - - - - -


# - - - - - - IMPORT FILES AND GRANT ACCESS - - - - - 
COPY . /app
WORKDIR /app
RUN useradd -ms /bin/bash admin
RUN chown -R admin:admin /app
RUN chmod 755 /app
USER admin
# - - - - - - - - - - - - - - - - - - - - - - - - - -



# - - - - - - INSTALL NUPIE SAMPLER - - - - - - - - -
RUN curl https://sh.rustup.rs  -sSf | bash -s -- -y
# RUN apt -y install gcc
ENV PATH="$PATH:/home/admin/.cargo/bin"
#llvm, gfortran
RUN pip install 'nutpie[pymc]'
# - - - - - - - - - - - - - - - - - - - - - - - - - -


CMD ["python", "run_model.py"]