Disable lock=True in open_mfdataset when reading netCDF3 files

[shoyer]

This slows things down unnecessarily.

[pwolfram]

@shoyer, is this related to #800? If I recall correctly based on #793 there may be some issues with dask/multithreading without the lock. Am I missing something here?

It may make sense to just to set lock=False by default in https://github.com/pydata/xarray/blob/master/xarray/backends/api.py#L247 and if issues are encountered lock=True can be set by the user. Is this what you are thinking? If so, I can submit the super simple PR if that is helpful.

[shoyer]

The issues with multithreading without a lock come up only when using netCDF4/HDF5. With netCDF3, you don't need the lock when reading data.

I don't like changing to lock=False in all circumstances, because that will lead to segfaults and segfaults make for a bad new user experience! Instead, lock=None should do something smart.

[pwolfram]

@shoyer, makes sense. Are you thinking of something like the following pseudo code?

if lock is None:
    if type(file) is netCDF4/HDF5:
        lock=True
    else:
        lock=False

[pwolfram]

We could also just filter the other way too which is probably cleaner:

if lock is None and type(file) is netCDF3:
   lock=False

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity
If this issue remains relevant, please comment here; otherwise it will be marked as closed automatically

[jhamman]

@shoyer - this was addressed in the backend refactor this summer, no?

[shoyer]

Yes it was