Forrtl: error (200): program aborting due to control-C event (even after cores=1 setted)

Questions
1

Hello, I am using
conda 4.8.4
python 3.7.8
PyMC3 v3.8
Windows 10
Visual Studio Code

and I get

forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFEA63E3B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFED80B5F63 Unknown Unknown Unknown
KERNEL32.DLL 00007FFED9017BD4 Unknown Unknown Unknown
ntdll.dll 00007FFEDAF0CE51 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFEA63E3B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFED80B5F63 Unknown Unknown Unknown
KERNEL32.DLL 00007FFED9017BD4 Unknown Unknown Unknown
ntdll.dll 00007FFEDAF0CE51 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFEA63E3B58 Unknown Unknown Unknown
KERNELBASE.dll 00007FFED80B5F63 Unknown Unknown Unknown
KERNEL32.DLL 00007FFED9017BD4 Unknown Unknown Unknown
ntdll.dll 00007FFEDAF0CE51 Unknown Unknown Unknown

time to time.

Not always. I means sometimes it is ok, but the error comes out occasionally.

Here is the part of my code.

with mts_model:
     
            mu = pm.Normal('cmu', mu = AvrCmu, sd = StdCmu)
            sigma = pm.Normal('csigma', mu = AvrCsig, sd = StdCsig)
            nu = pm.Normal('nu', mu = Avrnu, sd = Stdnu)
            delta = pm.Normal('delta', mu = Avrdelta, sd = Stddelta)

#shallow copy 

            Pl = P[:]; Wtl = Wt[:]; Tcl = Tc[:]; timingsl = timings[:]  

           likelihood = pm.Normal('timingsl', mu= Wtl/X , sd= formula , observed=timingsl)
           trace = pm.sample(draws= 500, cores=1)

           map_cmu.value = map_estimate2['cmu']
           map_csig.value = map_estimate2['csigma']
           map_nu.value = map_estimate2['nu']
           map_del.value = map_estimate2['delta']

I do the shallow copy here, because sometimes my data is added simultaneously with likelihood being updated.

Here i use multiprocessing to get data from certain experiment in real time and do the likelihood updates simultaneously.

So here is the deal, I found out related solution to this problem, and added “cores = 1” to my sample and it made the code work time to time but still I get the same problem often.

Thank you. Looking forward to your reply.

2

Could it be that you are running from terminal and sometimes you do Ctrl-C and terminated the program? (maybe you are doing something else while the sample is running and you are trying to copy and paste something)?

1 Like
3

Well, I use processing program at the same time, to get some data from it in real time. So I use socket to connect those two.

and I run this sample code from terminal and sometime I do Ctrl-C and terminate the program
but that error came out all the sudden while program was running, which means i did not hit the Ctrl-C at the time I faced that error.

So to be clear, I run certain program on the processing and conduct certain task on it.
Therefore, I get the data from it in real time while the sample is running (by multiprocessing).

and the data is variable named “timings” which I use for “observed” likelihood.
So to prevent it being affected by those data’s update, I did shallow copy before i use it as parameters for likelihood as you can see on my code.

Is there anything I should be care of?

4

I see, instead of shallow copy, you should look into using theano.shared

1 Like
5

Thanks for answering,
I am sorry, I did not quite understand what you mean.

The variables that I do shallow copy are the data that I get from processing.
So those variables are just actual values like time, or numbers of trial… they are not updated by pm.Normal function or else. I just use those copied values as parameters in likelihood parameters for calculating mu or sigma.

Is there any specific reason I need to use theano.shared? I’ve read document for theano but could not catch the sense the need of using it. Thank you very much.

I thought I found out the solution but it was not the answer. So I deleted the reply below

7

Well using theano.shared would allow you to compile the model once, and when data comes in you just update the shared variables and the model is updated, so I think it will be useful for you as you avoid recompiling the model and should save some time and compute.

I am not sure if it is related to your issue tho - this is something I have not seen before unfortunately.

2 Likes
8

A few observations to help with debugging - forrtl and libifcoremd.dll point to a Fortran error - is pymc3 still using Fortran for anything?

I’m pretty sure doing a copy, shallow or otherwise, on data that might be getting updated isn’t thread safe. That lowering the number of cores improved performance supports that - maybe look into thread safe practices

EDIT: just to clarify a bit, a deep copy would be thread safe if the updates are stopped during the copy process, which might be the case depending on your socket design. A shallow copy won’t ever be thread safe if there are pointers remaining. I had thought maybe the latter was your problem, but looking again at where the copy is I think you need to lock the original variables while the copy is happening.

2 Likes