I’m developing a project which will potentially generate lots of pymc3 fitting tasks which I want setup as celery tasks to queue and handle. Problem is as soon as I import pymc3 into a celery worker code the worker exits.
This is even without any actual pymc3 code included:
[2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-4' pid:7952 exited with 'exitcode 1' [2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-3' pid:15760 exited with 'exitcode 1' [2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-1' pid:15592 exited with 'exitcode 1' [2017-08-06 12:33:50,765: ERROR/MainProcess] Process 'Worker-2' pid:13536 exited with 'exitcode 1'
If I comment out the import of pymc3 the worker runs fine:
import pandas as pd #import pymc3 as pm import numpy as np import matplotlib.pyplot as plt import patsy as pt import seaborn as sns from theano import shared
Found this which seems to be the same/similar problem but no resolution on this one:
And minimal code to reproduce (saved as
from celery import Celery app = Celery('tasks', broker='amqp://') import pymc3 as pm @app.task() def add(x, y): return x + y if __name__ == '__main__': app.start()
Launch celery worker with
celery worker -A testcelery.app --loglevel=info