Using text backend raises sampler stats error

Questions
7

Your back of the envelope math got me curious, so I went back and looked at my logs, and it looks like I had used text from the get-go - I was assuming that I would have memory issues if I didn’t - and that I was hitting memory errors here:

    ris = trace.get_values('ris_level_offset', burn=1000, thin=2)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pymc3/backends/base.py", line 245, in get_values
    for chain in chains]
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pymc3/backends/text.py", line 125, in get_values
    self._load_df()
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pymc3/backends/text.py", line 104, in _load_df
    self.df = pd.read_csv(self.filename)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 401, in _read
    data = parser.read()
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 957, in read
    df = DataFrame(col_dict, columns=columns, index=index)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/frame.py", line 266, in __init__
    mgr = self._init_dict(data, index, columns, dtype=dtype)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/frame.py", line 402, in _init_dict
    return _arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/frame.py", line 5408, in _arrays_to_mgr
    return create_block_manager_from_arrays(arrays, arr_names, axes)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/internals.py", line 4262, in create_block_manager_from_arrays
    blocks = form_blocks(arrays, names, axes)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/internals.py", line 4331, in form_blocks
    float_blocks = _multi_blockify(float_items)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/internals.py", line 4408, in _multi_blockify
    values, placement = _stack_arrays(list(tup_block), dtype)
  File "/home/ubuntu/.virtualenvs/selfmade/local/lib/python2.7/site-packages/pandas/core/internals.py", line 4451, in _stack_arrays
    stacked = np.empty(shape, dtype=dtype)
MemoryError

I can try without using the text backend to see if I actually do get memory errors or if I was barking up the wrong tree.

As for numpy/scipy - I’ve installed it on even smaller instance types. This trick has come in handy.