PSA: File extension of netCDF files matters

Questions version agnostic
1

This is just a heads up about some weird behavior I encountered with reading netCDF files using ArviZ. I want to document it somewhere in case I run into the same problem in the future.

The file must have the extension .nc in order to be read by az.from_netcdf(). If the extension is .netcdf, then you will get the following error: ValueError: cannot guess the engine, try passing one explicitly. I could not locate where this error is coming from, but this change in file name fixed it.

I’m curious to know if anyone else has run into this issue or can replicate it (there is a chance that I’m doing something else wrong and this happened to fix it).

package    version
---        ---
Python     3.9.2
PyMC3      3.11.1
ArviZ      0.11.2
xarray     0.18.0
libnetcdf  4.8.0
netcdf4    1.5.6

Edit: I have updated the original post with the version of ‘xarray’ because, through the debugging outlined below, that turned out to be the cause.

2 Likes
2

That’s curious, FWIW I generally have no issues opening files with a .netcdf extension

e.g. az.from_netcdf(os.path.join(*dir_traces, f'{fn}.netcdf'))

MacOS 10.15.7

arviz>=0.11.0
pymc3==3.9.3
netCDF4==1.5.5.1

via brew:
netcdf: stable 4.7.4

1 Like
3

Thanks for you input. I think this may come from a change in netcdf or a related library because I’ve only had this issue recently. I should have noted that in the original post.

1 Like
4

Yeah, I see you’re using libnetcdf 4.8.0. If you can spare the time to downgrade and test, please do :slight_smile:

1 Like
5

I thought that would be the issue, but it doesn’t seem like it is. I tried with libnetcdf=4.7.4 and netcdf4=1.5.5.1 and only files with extension “.nc” worked. Below I have included some more information from the error about where it is being thrown.

store_spec = '/private/var/folders/r4/qpcdgl_14hbd412snp1jnv300000gn/T/pytest-of-admin/pytest-27/test_read_advi_cache0/inference-data.nd'

    def guess_engine(store_spec):
        engines = list_engines()
    
        for engine, backend in engines.items():
            try:
                if backend.guess_can_open and backend.guess_can_open(store_spec):
                    return engine
            except Exception:
                warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
    
>       raise ValueError("cannot guess the engine, try passing one explicitly")
E       ValueError: cannot guess the engine, try passing one explicitly

/usr/local/Caskroom/miniconda/base/envs/speclet-TEST/lib/python3.9/site-packages/xarray/backends/plugins.py:112: ValueError

It looks like it is coming from xarray (sorry for not noticing that earlier). I’m using xarray=0.18.0. Which version do you have?

1 Like
6

Ah cool, this bug hunting is never simple, so thanks for checking!

The environment I quoted above has xarray==0.16.2, so that might be a source of the difference…

2 Likes
7

That was the issue, I’ll update the original post. Thanks for your help @jonsedar!

1 Like
8

Cool, thanks for the PSA! Might be worth filing an issue with arviz so it can be addressed

1 Like
9

There is already an issue and PR about this at xarray: Engine is no longer inferred for filenames not ending in ".nc" · Issue #5295 · pydata/xarray · GitHub. My guess is that this will be fixed in the next bugfix version

2 Likes