@lucianopaz。Please help me solve the Windows multiprocessing issue


#1

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

Please save me from Windows10。✧⁺⸜(●˙▾˙●)⸝⁺✧


#2

@Abraxas2071, the images with the traceback are a nice and all but they don’t give me much context. I can see that you get the typical Broken Pipe Error that happens in windows when you try to sample with more than 1 core, and something goes wrong while the worker process are being spawned.

This problem happens a lot on windows and the cause of the problems are really diverse and depend much on how you are running your script. Are you running an interactive IPython session? Are you running the code from inside the spyder IDE? Are you running it from a windows Command Prompt?

To allow me to help you, you have to copy here the code that you ran and raised the error you pasted above. I would need a minimum working example to see what is going on.

Nevertheless, your traceback is showing an exception before the Broken Pipe, which could be the error that happened while the child process was being spawned. It looks like an unpickling error, as if some op cannot get compiled while it is being unpacked. Again, for me to be able to help you with this, add a minimum working example of the code. I would greatly appreciate if you could write it down using markdown, that way I can see it here well formatted and can copy easily.


#3

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

I don’t know how to use markdown…
Code is here:

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
# import seaborn as sns
import pymc3 as pm
import pandas as pd

N = 100
alpha_real = 2.5
beta_real = [0.9, 1.5]
eps_real = np.random.normal(0, 0.5, size=N)

X = np.array([np.random.normal(i, j, N) for i, j in zip([10, 2], [1, 1.5])])
X_mean = X.mean(axis=1, keepdims=True)
X_centered = (X - X.mean())  # / X.std(ddof=1)
y = alpha_real + np.dot(beta_real, X) + eps_real

if __name__ == '__main__':
    with pm.Model() as model_mlr:
        alpha_tmp = pm.Normal('alpha_tmp', mu=0, sd=10)
        beta = pm.Normal('beta', mu=0, sd=1, shape=2)
        epsilon = pm.HalfCauchy('epsilon', 5)

        mu = alpha_tmp + pm.math.dot(beta, X_centered)

        alpha = pm.Deterministic('alpha', alpha_tmp - pm.math.dot(beta, X_mean))

        y_pred = pm.Normal('y_pred', mu=mu, sd=epsilon, observed=y)

        trace_mlr = pm.sample(1000)
        pm.traceplot(trace_mlr)
        plt.show()

Pycharm、Miniconda、Win10


#5

Could you try to see if this code works?

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
# import seaborn as sns
import pymc3 as pm
import pandas as pd

N = 100
alpha_real = 2.5
beta_real = [0.9, 1.5]
eps_real = np.random.normal(0, 0.5, size=N)

X = np.array([np.random.normal(i, j, N) for i, j in zip([10, 2], [1, 1.5])])
X_mean = X.mean(axis=1, keepdims=True)
X_centered = (X - X.mean())  # / X.std(ddof=1)
y = alpha_real + np.dot(beta_real, X) + eps_real

with pm.Model() as model_mlr:
    alpha_tmp = pm.Normal('alpha_tmp', mu=0, sd=10)
    beta = pm.Normal('beta', mu=0, sd=1, shape=2)
    epsilon = pm.HalfCauchy('epsilon', 5)

    mu = alpha_tmp + pm.math.dot(beta, X_centered)

    alpha = pm.Deterministic('alpha', alpha_tmp - pm.math.dot(beta, X_mean))

    y_pred = pm.Normal('y_pred', mu=mu, sd=epsilon, observed=y)

if __name__ == '__main__':
    with model_mlr:
        trace_mlr = pm.sample(1000)
        pm.traceplot(trace_mlr)
        plt.show()

If it still does not work, does it work if you sample with 1 core like this: trace_mlr = pm.sample(1000, cores=1)?


#6

Another question

Can you run a simple stand alone theano code without errors? Something like the following:

import numpy as np
import theano
from theano import tensor as tt

x = tt.dmatrix('x')
y = tt.dmatrix('y')
z = tt.dot(x, y)
f = theano.function([x, y], z)

print(f(np.random.randn(2, 10), np.random.randn(10, 4)))

#7

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

In order not to delay learning the book《Bayesian Analysis with Python》,I uninstalled miniconda, otherwise I couldn’t run the code
So when I reinstall miniconda and input “conda install pymc3” for try your code
Oh GOD!I’m going crazy fast~o(╥﹏╥)o
Warning happen:

===========================================
I install / unstall over and over again “matplotlib”、“pymc3” by Pycharm or input “conda install / uninstall”
But the warning still exist…
找不到指定模块 mean is can’t find module


#8

The installation is still wrong.

Just to better understand, @Abraxas2071, you completely removed miniconda and all of the packages installed under it, then you re installed miniconda, and finally just ran conda install pymc3?

Then when you are trying to run your script, are you writing python.exe Global_Observer.py yourself or is PyCharm doing that for you? Maybe PyCharm is using a different python installation than the one done by conda.

Could you open a Command Prompt and type where python or where python.exe?


#9

Part 3 of this link describes how to select from inside PyCharm, the python executable that is installed by miniconda or anaconda. Then there is also this one here. Maybe they could help you


#10

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

Yes!Finally just~
I’m so sorry for my badly English~

Myself add the .py
However use PyCharm or miniconda’s “conda install / uninstall”, the wrong still exist

cmd ine


#11

China’s Internet has some restrictions。I can’t open the links


#12

Ok, so the problem is that you are not running the python executable that is installed by mini conda. I’ll copy paste from the links how to force pycharm to use the python installed by anaconda


#13

Or run your program line by line in the terminal - it’s much easier to learn and debug


#14

I’m copy pasting from one of the links, it will be formatted wrong because I’m on my phone:

I recently had to add an anaconda environment to already created conda environment and faced some issue. Thought I will explain this so others don’t have to go through the same thing.

I am using PyCharm community edition 2017.3.4 so if you are using any other version of PyCharm you will have to use your JetBrains tool skills to change the steps. Check this video out if you want to improve those skills. A JetBrains employee told me he intends to do a PyCharm version too.

Now the actual steps for this version

  • Click on Configure > Settings to open up settings in PyCharm
  • Search for “Project Interpreter”. My PyCharm looks like this

  • Click on Add local via the settings on the right side

  • Select “conda environment”
  • Click on “Existing environment” and navigate to the environment that you want to use. Note that you have to select the bin/python file inside the conda environment for PyCharm to be able to recognise the environment
  • Make sure to click the “Make available to all projects” if you want the interpreter to be used by multiple projects

  • Click ok and you are done

#15

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

The link’s method was used before open the question…But it still doesn’t work
In a long time ago, I reinstall packages many times(like “matplotlib” “pymc3” “pandas” “numpy”),It‘s’ working!Then warning: Windows multiprocessing issue。
So I uninstalled it while waiting for your reply, I don’t know why it doesn’t work again

These things are frustrating。 It’s already past 10 o 'clock in China time, I will wait for your reply in my sleepZZz…(¦3ꇤ[▓▓]


#16

@junpenglao
Thank you~
Don’t worry 。I have some basis to use PyCharm(我有一些使用Pycharm的基础,也通过它学习了Python和machine learning)
ZZz…(¦3ꇤ[▓▓]


#17

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

I’m awake~I’m waiting for you~(o゚▽゚)o


#18

Could you try to run this and tell me what gets printed out?

import sys, numpy, theano
print(sys.path)
print(numpy.__file__)
print(theano.__file__)

#19

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

I can’t run the code…Because——

==============================================
I uninstalled all of them(Python3.7 and miniconda,All uninstall) except Pycharm and reinstalled miniconda, This warning still exists
Now I can’t do anything except install Python3.7 separately。I can’t test any of your code
o(╥﹏╥)oDear Luciano。。。What shall I do??Help…Help…


#20

But your picture says that the project interpreter is the mini conda python.exe. It also says that numpy and theano are already installed.


#21

@lucianopaz
Pymc3’s charioteer, Bayesian’s faithful follower,and Sickness’s fighter,Our Dear Luciano, Glory to you!

I have one Good News and one Bad News。I know you love Bad News first,haahaa~
Bad News:I lost my bankcard,So I went to apply for a bank card,Sorry to have kept you waiting so long。
Good News:I reinstalled Python3.7 and choose it as the environment for Pycharm,But did’t work。So I choose miniconda again。
The miracle happened!It is working!We finally got down to the issue:

================== no “cores=1” =====================

================== have “cores=1” ===================

==================

====================

==================

====================

Our Dear Luciano。What’s your opinion?;ヾ(◍°∇°◍)ノ゙