I’m running a large Pymc3 model which is doing marginalisation over multiple submodels. That marginalisation code ostensibly works - as I can run it on simple single models fine. However, when scaling it up to include other independent models, the model initialises perfectly fine and then fails with a seg fault every time.
Some info: MacBook Pro running Catalina. Python 3.7. Pymc3 3.8. Theano 1.0.4.
The model is complex so I can’t really post a working example (but the code is in the init_model function here: https://github.com/hposborn/MonoTools/blob/master/MonoFit.py)
It involves looping over set values of some discontinous variable, computing a model and a logprob, marginalising over those submodels to create a pm.Potential for the marginalised model.
It only fails when I include another independent model, for which another pm.Potential is called afterwards. I believe they are initialised correctly, again because the individual models work fine when run independently.
Can anyone help me out interpreting these error messages? I honestly have no idea whether the problem is in the marginalisation, or in my GCC/environment variables (e.g. the dylib), or something like that.
This is the error output:
Sampling 4 chains, 0 divergences: 0%| | 0/1660 [00:00<?, ?draws/s][My-Mac:56208] *** Process received signal *** [My-Mac:56208] Signal: Segmentation fault: 11 (11) [My-Mac:56208] Signal code: Address not mapped (1) [My-Mac:56208] Failing at address: 0x110 [My-Mac:56208] [ 0] 0 libsystem_platform.dylib 0x00007fff65f0d42d _sigtramp + 29 [My-Mac:56208] [ 1] 0 ??? 0x00007ffee17ee450 0x0 + 140732681610320 [My-Mac:56208] [ 2] 0 libBLAS.dylib 0x00007fff2aa19de5 cblas_dgemv + 853 [My-Mac:56208] [ 3] 0 libBLAS.dylib 0x00007fff2aa19c19 cblas_dgemv + 393 [My-Mac:56208] [ 4] 0 libBLAS.dylib 0x00007fff2aa1ce66 DGEMV + 136 [My-Mac:56208] [ 5] 0 md1ee011f7a18103231f701966a7f0d3b5e 0x0000000180ec3563 _ZL95__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47e_executorPN12_GLOBAL__N_186__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47eE + 6803 [My-Mac:56208] [ 6] 0 lazylinker_ext.so 0x000000012e331fdd _ZL6c_callP11CLazyLinkerli + 445 [My-Mac:56208] [ 7] 0 lazylinker_ext.so 0x000000012e331853 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1955 [My-Mac:56208] [ 8] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [ 9] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [10] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [11] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [12] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [13] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [14] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [15] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [16] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [17] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [18] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [19] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [20] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [21] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [22] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [23] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [24] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [25] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [26] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [27] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [28] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56208] [29] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] *** Process received signal *** [My-Mac:56207] *** Process received signal *** [My-Mac:56208] *** End of error message *** [My-Mac:56209] Signal: Segmentation fault: 11 (11) [My-Mac:56209] Signal code: Address not mapped (1) [My-Mac:56209] Failing at address: 0x110 [My-Mac:56209] [ 0] 0 libsystem_platform.dylib 0x00007fff65f0d42d _sigtramp + 29 [My-Mac:56209] [ 1] 0 ??? 0x00007ffee17ee450 0x0 + 140732681610320 [My-Mac:56209] [ 2] 0 libBLAS.dylib 0x00007fff2aa19de5 cblas_dgemv + 853 [My-Mac:56209] [ 3] 0 libBLAS.dylib 0x00007fff2aa19c19 cblas_dgemv + 393 [My-Mac:56209] [ 4] 0 libBLAS.dylib 0x00007fff2aa1ce66 DGEMV + 136 [My-Mac:56209] [ 5] 0 md1ee011f7a18103231f701966a7f0d3b5e 0x0000000180ec3563 _ZL95__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47e_executorPN12_GLOBAL__N_186__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47eE + 6803 [My-Mac:56209] [My-Mac:56207] Signal: Segmentation fault: 11 (11) [My-Mac:56207] Signal code: Address not mapped (1) [My-Mac:56207] Failing at address: 0x110 [My-Mac:56210] *** Process received signal *** [ 6] [My-Mac:56207] [ 0] 0 lazylinker_ext.so 0x000000012e331fdd _ZL6c_callP11CLazyLinkerli + 445 [My-Mac:56209] [ 7] 0 lazylinker_ext.so 0x000000012e331853 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1955 0 libsystem_platform.dylib 0x00007fff65f0d42d _sigtramp + 29 [My-Mac:56209] [My-Mac:56207] [ 8] [ 1] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [ 9] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [10] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [11] 0 ??? 0x00007ffee17ee450 0x0 + 140732681610320 [My-Mac:56207] [My-Mac:56210] Signal: Segmentation fault: 11 (11) [My-Mac:56210] Signal code: Address not mapped (1) [My-Mac:56210] Failing at address: 0x110 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [ 2] [My-Mac:56209] [12] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [13] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] 0 libBLAS.dylib 0x00007fff2aa19de5 cblas_dgemv + 853 [ 0] [My-Mac:56207] [ 3] 0 libBLAS.dylib 0x00007fff2aa19c19 cblas_dgemv + 393 [My-Mac:56209] [My-Mac:56207] [ 4] [14] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 libBLAS.dylib 0x00007fff2aa1ce66 DGEMV + 136 [My-Mac:56209] [15] 0 libsystem_platform.dylib 0x00007fff65f0d42d _sigtramp + 29 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56209] [My-Mac:56210] [16] [ 5] [ 1] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [17] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [18] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [19] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [20] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 md1ee011f7a18103231f701966a7f0d3b5e 0x0000000180ec3563 _ZL95__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47e_executorPN12_GLOBAL__N_186__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47eE + 6803 0 ??? 0x00007ffee17ee450 0x0 + 140732681610320 [My-Mac:56209] [My-Mac:56210] [My-Mac:56207] [ 2] [ 6] [21] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 libBLAS.dylib 0x00007fff2aa19de5 cblas_dgemv + 853 [My-Mac:56209] [My-Mac:56210] [22] [ 3] 0 lazylinker_ext.so 0x000000012e331fdd _ZL6c_callP11CLazyLinkerli + 445 [My-Mac:56207] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [23] 0 libBLAS.dylib 0x00007fff2aa19c19 cblas_dgemv + 393 [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [ 7] [ 4] [My-Mac:56209] 0 lazylinker_ext.so 0x000000012e331853 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1955 0 libBLAS.dylib 0x00007fff2aa1ce66 DGEMV + 136 [My-Mac:56210] [ 5] [24] [My-Mac:56207] [ 8] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [25] 0 md1ee011f7a18103231f701966a7f0d3b5e 0x0000000180ec3563 _ZL95__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47e_executorPN12_GLOBAL__N_186__struct_compiled_op_md1ee011f7a18103231f701966a7f0d3b5e515c577232063cea2c178350b9d47eE + 6803 [My-Mac:56210] [ 6] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 2_ + 1784 0 lazylinker_ext.so 0x000000012e331fdd _ZL6c_callP11CLazyLinkerli + 445 [My-Mac:56209] [My-Mac:56210] [My-Mac:56207] [26] [ 7] [ 9] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e331853 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1955 [My-Mac:56209] [My-Mac:56210] [27] [ 8] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [My-Mac:56210] [28] [ 9] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] [My-Mac:56207] [My-Mac:56210] [29] [10] [10] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56209] *** End of error message *** 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [11] [11] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [My-Mac:56207] [12] [12] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [13] [My-Mac:56207] [13] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [14] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [15] [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [14] [My-Mac:56207] [16] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [15] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [17] [16] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [18] [17] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [19] [18] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [My-Mac:56207] [19] [20] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [20] [My-Mac:56207] [21] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [21] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [22] [My-Mac:56210] [22] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [23] [23] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [24] [My-Mac:56210] [24] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [25] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [25] [My-Mac:56207] [26] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [26] [My-Mac:56207] [27] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [27] [My-Mac:56207] [28] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] [28] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56207] [My-Mac:56210] [29] [29] 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 0 lazylinker_ext.so 0x000000012e3317a8 _ZL13lazy_rec_evalP11CLazyLinkerlP7_objectS2_ + 1784 [My-Mac:56210] *** End of error message *** [My-Mac:56207] *** End of error message ***