ERROR (theano.gof.opt) & slowness in execution in sampling

Questions
1

Dear All,
I finished writing my forward model for pymc model using theano. This is my preliminary project and once it’s get done accurately, I can move forward to my main project. I got no error in the section of pymc_model but further sampling takes too much time and return with error. Considering my current knowledge about PyMc3, I have no clue how can I improve my model. Here I am posting the error message and also attaching my code.
Description: Main_code along with the code for forward model and pymc model. Please run Main_Code.
I am looking for your kind suggestions. Please help me to improve the code pymc_forward_model.
Thanking you

Sincerely,
mahak

Attached File

Main_Code.py (1.6 KB)
Forward_Model.py (3.9 KB)
pymc_forward_model.py (4.2 KB)

Output Error
ERROR (theano.gof.opt): Optimization failure due to: local_mul_zero
ERROR (theano.gof.opt): node: Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of -1.0}, Elemwise{add,no_inplace}.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\gof\opt.py”, line 2034, in process_node
replacements = lopt.transform(node)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\opt.py”, line 5811, in local_mul_zero
value = get_scalar_constant_value(i)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py”, line 511, in get_scalar_constant_value
v.owner.op.perform(v.owner, const, ret)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\elemwise.py”, line 718, in perform
super(Elemwise, self).perform(node, inputs, output_storage)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\gof\op.py”, line 778, in perform
“Did you used Theano flags mode=FAST_COMPILE?”
theano.gof.utils.MethodNotDefined: (‘perform’, <class ‘theano.tensor.elemwise.Elemwise’>, ‘Elemwise’, ‘Did you used Theano flags mode=FAST_COMPILE? You can use optimizer=fast_compile instead.’)

Sequential sampling (2 chains in 1 job)
HamiltonianMC: [sigma, Tens, Dip_s, Stk_s, Wid, DIP, Azim, Depth, N, E, Len]
0%| | 103/52000 [00:30<1:19:03, 10.94it/s]
Traceback (most recent call last):
File “invert_Okada.py”, line 92, in
trace = pm.sample(50000, tune=2000, init=‘none’, step=step, cores=1)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\sampling.py”, line 459, in sample
trace = _sample_many(**sample_args)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\sampling.py”, line 505, in _sample_many
step=step, random_seed=random_seed[i], **kwargs)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\sampling.py”, line 549, in _sample
for it, strace in enumerate(sampling):
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\tqdm_tqdm.py”, line 1022, in iter
for obj in iterable:
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\sampling.py”, line 645, in _iter_sample
point, states = step.step(point)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\step_methods\arraystep.py”, line 247, in step
apoint, stats = self.astep(array)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\step_methods\hmc\base_hmc.py”, line 135, in astep
self.potential.raise_ok(self._logp_dlogp_func._ordering.vmap)
File “C:\Users\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\step_methods\hmc\quadpotential.py”, line 231, in raise_ok
raise ValueError(’\n’.join(errmsg))
ValueError: Mass matrix contains zeros on the diagonal.
The derivative of RV sigma_log__.ravel()[0] is zero.
The derivative of RV Tens_interval__.ravel()[0] is zero.
The derivative of RV Dip_s_interval__.ravel()[0] is zero.
The derivative of RV Stk_s_interval__.ravel()[0] is zero.
The derivative of RV Wid_interval__.ravel()[0] is zero.
The derivative of RV DIP_interval__.ravel()[0] is zero.
The derivative of RV Azim_interval__.ravel()[0] is zero.
The derivative of RV Depth_interval__.ravel()[0] is zero.
The derivative of RV N_interval__.ravel()[0] is zero.
The derivative of RV E_interval__.ravel()[0] is zero.
The derivative of RV Len_interval__.ravel()[0] is zero.

2

Your model could be optimized by vectorizing some operation (the nested function is hard to read and whenever you are calling the same function on different inputs, there is a high chance you can vectorized those and make it more compact), but there is nothing inherently wrong of the way you write down the model. The big red flag is more the way you call the sampling:

with basic_model:
    step=pm.HamiltonianMC()
    trace = pm.sample(50000, tune=2000, init='none', step=step, cores=1)

Which should be:

with basic_model:
    trace = pm.sample(1000, tune=2000)

Always start with the default sampling option!

And if this still gives you error, you will need to optimized the model building part.

3

Can you please explain little bit this line. I have made many test, also using NUTS (default) sampler but I am always getting almost same error as following:

ERROR (theano.gof.opt): Optimization failure due to: local_mul_zero
ERROR (theano.gof.opt): node: Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of -1.0}, Elemwise{add,no_inplace}.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "C:\Users\singh.CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\gof\opt.py", line 2034, in process_node
    replacements = lopt.transform(node)
  File "C:\Users\singh.CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\opt.py", line 5811, in local_mul_zero
    value = get_scalar_constant_value(i)
  File "C:\Users\singh.CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py", line 511, in get_scalar_constant_value
    v.owner.op.perform(v.owner, const, ret)
  File "C:\Users\singh.CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\elemwise.py", line 718, in perform
    super(Elemwise, self).perform(node, inputs, output_storage)
  File "C:\Users\singh.CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\gof\op.py", line 778, in perform
    "Did you used Theano flags mode=FAST_COMPILE?"
theano.gof.utils.MethodNotDefined: ('perform', <class 'theano.tensor.elemwise.Elemwise'>, 'Elemwise', 'Did you used Theano flags mode=FAST_COMPILE? You can use optimizer=fast_compile instead.')
4

Dear All,
I have improved my forward model and now completely removed the loop. The code is simpler and smaller now. But still I am having hard time to understand the use of theano. I am attaching my codes again below where Forward_Model is my numerical code to calculate the observed data, pymc_forward_model is the one I am trying to adapt for pymc3 model and both codes are called in the Main_Code.

Your help to improve pymc_forward_model will be very beneficial for me. Please note that I am newbie to pymc3

Thanking You very much
Sincerely
Mahak

Forward_Model.py (4.2 KB)
Main_Code.py (1.5 KB)
pymc_forward_model.py (3.8 KB)

import pymc3 as pm
import pymc_forward_model as PFM
My_model = pm.Model()
with My_model:
E=pm.Normal(‘E’, mu=0, sd=500)
N=pm.Normal(‘N’, mu=0, sd=500)
Depth=pm.Uniform(‘Depth’,lower=-6000,upper=-1000)
Azim=pm.Uniform(‘Azim’,lower=0,upper=360)
Bounded_DIP = pm.Bound(pm.Normal, lower=10, upper=89.9)
DIP = Bounded_DIP(‘DIP’, mu=50, sd=3.0)
Len=pm.Uniform(‘Len’,lower=100,upper=10000)
Wid=pm.Uniform(‘Wid’,lower=100,upper=10000)
Bounded_Tens = pm.Bound(pm.Normal, lower=0, upper=10)
Tens = Bounded_Tens(‘Tens’, mu=0, sd=0.5)
sigma=1
unknowns = [E,N,Depth,Azim,DIP,Len,Wid,Stk_s1,Dip_s1,Tens]
mu1 = PFM.fun1(unknowns, x,y,z, Poisson)
Y_obs = pm.Normal(‘Y_obs’, mu=mu1, sd=sigma, observed=U)

Error:

C:\CT\Desktop\Discourse>python Main_Code.py
Traceback (most recent call last):
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\type.py”, line 269, in dtype_specs
}[self.dtype]
KeyError: ‘object’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py”, line 246, in constant
ttype = TensorType(dtype=x_.dtype, broadcastable=bcastable)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\type.py”, line 51, in init
self.dtype_specs() # error checking is done there
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\type.py”, line 272, in dtype_specs
% (self.class.name, self.dtype))
TypeError: Unsupported dtype for TensorType: object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py”, line 194, in as_tensor_variable
return constant(x, name=name, ndim=ndim)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py”, line 266, in constant
raise TypeError(“Could not convert %s to TensorType” % x, type(x))
TypeError: (‘Could not convert [Elemwise{add,no_inplace}.0 Elemwise{add,no_inplace}.0\n Elemwise{add,no_inplace}.0] to TensorType’, <class ‘numpy.ndarray’>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “Main_Code.py”, line 41, in
Y_obs = pm.Normal(‘Y_obs’, mu=mu1, sd=sigma, observed=U)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\distributions\distribution.py”, line 41, in new
dist = cls.dist(*args, **kwargs)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\distributions\distribution.py”, line 52, in dist
dist.init(*args, **kwargs)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\pymc3\distributions\continuous.py”, line 431, in init
self.mean = self.median = self.mode = self.mu = mu = tt.as_tensor_variable(mu)
File “C:\CT\AppData\Local\Continuum\anaconda3\lib\site-packages\theano\tensor\basic.py”, line 200, in as_tensor_variable
raise AsTensorError(“Cannot convert %s to TensorType” % str_x, type(x))
theano.tensor.var.AsTensorError: (‘Cannot convert [Elemwise{add,no_inplace}.0 Elemwise{add,no_inplace}.0\n Elemwise{add,no_inplace}.0] to TensorType’, <class ‘numpy.ndarray’>)