Skip to content

clib: Add the virtualfile_out method for creating output virtualfile #3057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Feb 28, 2024
1 change: 1 addition & 0 deletions doc/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@ conversion of Python variables to GMT virtual files:
clib.Session.virtualfile_from_matrix
clib.Session.virtualfile_from_vectors
clib.Session.virtualfile_from_grid
clib.Session.virtualfile_out


Low level access (these are mostly used by the :mod:`pygmt.clib` package):
Expand Down
73 changes: 73 additions & 0 deletions pygmt/clib/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -1609,6 +1609,79 @@ def virtualfile_from_data( # noqa: PLR0912

return file_context

@contextlib.contextmanager
def virtualfile_out(
self, kind: Literal["dataset", "grid"] = "dataset", fname: str | None = None
):
"""
Create a virtual file or an actual file for storing output data.

If ``fname`` is not given, a virtual file will be created to store the output
data into a GMT data container and the function yields the name of the virutal
file. Otherwise, the output data will be written into the specified file and the
function simply yields the actual file name.

Parameters
----------
kind
The data kind of the virtual file to create. Valid values are ``"dataset"``
and ``"grid"``. Ignored if ``fname`` is specified.
fname
The name of the actual file to write the output data. No virtual file will
be created.

Yields
------
vfile : str
Name of the virtual file or the actual file.

Examples
--------
>>> from pathlib import Path
>>> from pygmt.clib import Session
>>> from pygmt.datatypes import _GMT_DATASET, _GMT_GRID
>>> from pygmt.helpers import GMTTempFile
>>>
>>> # Create a virtual file for storing the output table.
>>> with GMTTempFile(suffix=".txt") as tmpfile:
... with open(tmpfile.name, mode="w") as fp:
... print("1.0 2.0 3.0 TEXT", file=fp)
... with Session() as lib:
... with lib.virtualfile_out(kind="dataset") as vouttbl:
... lib.call_module("read", f"{tmpfile.name} {vouttbl} -Td")
... ds = lib.read_virtualfile(vouttbl, kind="dataset")
>>> isinstance(ds.contents, _GMT_DATASET)
True
>>>
>>> # Create a virtual file for storing the output grid.
>>> with Session() as lib:
... with lib.virtualfile_out(kind="grid") as voutgrd:
... lib.call_module("read", f"@earth_relief_01d_g {voutgrd} -Tg")
... outgrd = lib.read_virtualfile(voutgrd, kind="grid")
>>> isinstance(outgrd.contents, _GMT_GRID)
True
>>>
>>> # Write data to file without creating a virtual file
>>> with GMTTempFile(suffix=".nc") as tmpfile:
... with Session() as lib:
... with lib.virtualfile_out(
... kind="grid", fname=tmpfile.name
... ) as voutgrd:
... lib.call_module("read", f"@earth_relief_01d_g {voutgrd} -Tg")
... assert voutgrd == tmpfile.name
... assert Path(voutgrd).stat().st_size > 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this into a new test_clib_virtualfile_out.py file? Two reasons:

  1. There might be other data types like GMT_IMAGE and GMT_CUBE in the future, and the doctest would get long.
  2. Might want to test the output of the virtualfiles more properly using pygmt.info or pygmt.grdinfo.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this into a new test_clib_virtualfile_out.py file? Two reasons:

  1. There might be other data types like GMT_IMAGE and GMT_CUBE in the future, and the doctest would get long.

The main purpose of these doctests are to show how to use this method and the outpu virtual file in other modules. So I think we can only keep the dataset one only and remove all others.

  1. Might want to test the output of the virtualfiles more properly using pygmt.info or pygmt.grdinfo.

Actually we can't. Because the virtual file is an output (i.e., GMT_OUT), it can't be passed as an input to info/grdinfo. I.e., lib.call_module("info", f"{vouttbl}") won't work.

Copy link
Member

@weiji14 weiji14 Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think we can only keep the dataset one only and remove all others.

You can move the grid one to test_clib_virtualfile_out.py

3. Might want to test the output of the virtualfiles more properly using pygmt.info or pygmt.grdinfo.

Actually we can't. Because the virtual file is an output (i.e., GMT_OUT), it can't be passed as an input to info/grdinfo. I.e., lib.call_module("info", f"{vouttbl}") won't work.

Ah ok, maybe not for the virtualfile then. But we should test that saving to a (non-virtual) file works (when fname is used), and try loading from there (i.e. roundtrip). Again, can put this in test_clib_virtualfile_out.py

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this into a new test_clib_virtualfile_out.py file? Two reasons:

  1. There might be other data types like GMT_IMAGE and GMT_CUBE in the future, and the doctest would get long.

The main purpose of these doctests are to show how to use this method and the outpu virtual file in other modules. So I think we can only keep the dataset one only and remove all others.

In commit 9108917, I've removed the doctest for grid and kept the doctests for dataset and fname. I also realized that these doctests are almost the same as the ones in read_virtualfile. This makes sense, because read_virtualfile was implemented to read the data from an output virtualfile. So, I also simplify the doctests in read_virtualfile using the newly added virtualfile_out method.

In other words, we probably need to move the doctests of read_virtualfile into a separate test file in the future.

Copy link
Member Author

@seisman seisman Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think we can only keep the dataset one only and remove all others.

You can move the grid one to test_clib_virtualfile_out.py

As mentioned in the above comment, the grid one is already doctested in the read_virtualfile method.

  1. Might want to test the output of the virtualfiles more properly using pygmt.info or pygmt.grdinfo.

Actually we can't. Because the virtual file is an output (i.e., GMT_OUT), it can't be passed as an input to info/grdinfo. I.e., lib.call_module("info", f"{vouttbl}") won't work.

Ah ok, maybe not for the virtualfile then. But we should test that saving to a (non-virtual) file works (when fname is used), and try loading from there (i.e. roundtrip). Again, can put this in test_clib_virtualfile_out.py

Done in 888f1c9 but not in a separate test file.

"""
if fname is not None: # Yield the actual file name.
yield fname
else: # Create a virtual file for storing the output data.
# Determine the family and geometry from kind
family, geometry = {
"grid": ("GMT_IS_GRID", "GMT_IS_SURFACE"),
"dataset": ("GMT_IS_DATASET", "GMT_IS_PLP"),
}[kind]
with self.open_virtualfile(family, geometry, "GMT_OUT", None) as vfile:
yield vfile

def read_virtualfile(
self, vfname: str, kind: Literal["dataset", "grid", None] = None
):
Expand Down