I’m developing a project which will potentially generate lots of pymc3 fitting tasks which I want setup as celery tasks to queue and handle. Problem is as soon as I import pymc3 into a celery worker code the worker exits.
This is even without any actual pymc3 code included:
[2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-4' pid:7952 exited with 'exitcode 1'
[2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-3' pid:15760 exited with 'exitcode 1'
[2017-08-06 12:33:49,948: ERROR/MainProcess] Process 'Worker-1' pid:15592 exited with 'exitcode 1'
[2017-08-06 12:33:50,765: ERROR/MainProcess] Process 'Worker-2' pid:13536 exited with 'exitcode 1'
If I comment out the import of pymc3 the worker runs fine:
import pandas as pd
#import pymc3 as pm
import numpy as np
import matplotlib.pyplot as plt
import patsy as pt
import seaborn as sns
from theano import shared
Found this which seems to be the same/similar problem but no resolution on this one:
And minimal code to reproduce (saved as
from celery import Celery
app = Celery('tasks', broker='amqp://')
import pymc3 as pm
def add(x, y):
return x + y
if __name__ == '__main__':
Launch celery worker with
celery worker -A testcelery.app --loglevel=info