Save and load in binary, compatible with NumPy/Matlab and others

[certik]

• NumPy: https://numpy.org/devdocs/reference/generated/numpy.lib.format.html
• Matlab: https://www.mathworks.com/help/matlab/import_export/mat-file-versions.html
• SciPy code to load/write MAT files: https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html
• Julia library to read/write MAT files: https://github.com/JuliaIO/MAT.jl
• Specification of MAT file format: https://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

First requested here.

[milancurcic]

This seems useful and in scope. I often work with MAT files (of various versions) from colleagues and I use SciPy.io loadmat and savemat.

For my own interoperable binary data between Fortran and Python, I use NetCDF. I don't think any of the language-specific binary formats will beat it in terms of features, performance, or stability. Likewise for HDF5 which is suitable for unstructured data.

[certik]

Both NetCDF and HDF5 are great. The only issue with HDF5 is that there is literally only one library that can read and write it and it's not that easy to build and ship. It's not easy to write a writer in pure Fortran, as an example. While it is easy for the .npy NumPy array format, I've done it in the past, although I can't find the code right now. :(

So that makes me hesitant to just depend on HDF5. However, it is worth investigating what would it take to just support a very small subset of HDF5, say for writing a set of double precision arrays. It might not be that difficult to write a writer for just such a small subset in pure Fortran. Here is the format: https://portal.hdfgroup.org/display/HDF5/File+Format+Specification

The huge advantage of that would be no dependency on the hdf5 library, and using a widely supported format.

[Beliavsky]

There is

NPY for Fortran: allows saving numerical Fortran arrays in Numpy's .npy or .npz format, by MRedies

which I have not tried.

[MarDiehl]

There is already an HDF5 writer/reader which looks promising: https://github.com/geospace-code/h5fortran. I uses the Fortran bindings of the C library.
I think it is reasonable to keep HDF5 support out of stdlib, it is neither part of the C nor the python standard library.

[awvwgk]

I got the basic structure for reading and writing npy files implemented in #581. Needs some polishing, especially the reading, and much more unit tests to cover all possible errors the loading can encounter.

[TejasAvinashShetty]

libnpy seems to be a library that provides simple routines for saving a C or Fortran array to a data file using NumPy's own binary format.
Please see https://scipy-cookbook.readthedocs.io/items/InputOutput.html

Not my idea See first CAZT's comment on CAZT's stackoverflow answer

[jvdp1]

There is already an HDF5 writer/reader which looks promising: https://github.com/geospace-code/h5fortran. I uses the Fortran bindings of the C library. I think it is reasonable to keep HDF5 support out of stdlib, it is neither part of the C nor the python standard library.

I agree with @MarDiehl . I recently used @scivision 's h5fortran and found it great and really easy to use. Therefore, I also think reasonable to keep HDF5 support out of stdlib for the moment.

[awvwgk]

How do we want to handle the npz format? It is a zip archive with npy files. Probably, we have to develop a general interface for interacting with compressed archives first.

For the mat format I found a specification of the layout (linked in description at the top), should be straight-forward to code up, but I don't think I have a matlab version I could use to verify it, but I could try SciPy.

[ivan-pi]

For the mat format I found a specification of the layout (linked in description at the top), should be straight-forward to code up, but I don't think I have a matlab version I could use to verify it, but I could try SciPy.

Was your idea to implement the reader/writer entirely in Fortran based upon the PDF document, or call into the MATLAB C API to Read MAT-File Data? The latter requires the client has the libmat shared run-time library located in matlabroot/bin/arch.

[awvwgk]

I was reading the specs, sounds easy enough to implement this from scratch and verify using SciPy. Unfortunately, the data can be compressed, and we need an interface to zlib or similar first.

Having the possibility to dynamically load a library with dlopen in case the matlab runtime libraries are around would be another option. However, than we first need an interface for dynamic loading.

[arjenmarkus]

My dynlib module in https://sourceforge.net/p/flibs/svncode/HEAD/tree/trunk/src/dynlib/ could serve as a starting point. Op zo 12 dec. 2021 om 17:03 schreef Sebastian Ehlert < ***@***.***>:

…

I was reading the specs, sounds easy enough to implement this from scratch and verify using SciPy. Unfortunately, the data can be compressed, and we need an interface to zlib or similar first. Having the possibility to dynamically load a library with dlopen in case the matlab runtime libraries are around would be another option. However, than we first need an interface for dynamic loading. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#486 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR7K6GAMFCJOU6EIQWTUQTBVJANCNFSM5CM7FYQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[adenchfi]

I see stdlib has save_npy and load_npy functionality! I tested it out and it works great! I was wondering, if possible, if dim(:,:,:,:) arrays could also be supported. I only see interfaces up to rank-3.