Skip to content

Commit f26e944

Browse files
jswhitjswhit
authored
Merge pull request #1295 from ZedThree/add-complex-number-support
Add complex number support
2 parents d022753 + fe653fe commit f26e944

File tree

14 files changed

+536
-127
lines changed

14 files changed

+536
-127
lines changed

‎.github/workflows/build_latest.yml

+2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ jobs:
1717
steps:
1818

1919
- uses: actions/checkout@v4
20+
with:
21+
submodules: true
2022

2123
- name: Set up Python ${{ matrix.python-version }}
2224
uses: actions/setup-python@v5

‎.github/workflows/build_master.yml

+2
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ jobs:
1414
steps:
1515

1616
- uses: actions/checkout@v4
17+
with:
18+
submodules: true
1719

1820
- name: Set up Python ${{ matrix.python-version }}
1921
uses: actions/setup-python@v5

‎.github/workflows/build_old.yml

+2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ jobs:
1717
steps:
1818

1919
- uses: actions/checkout@v4
20+
with:
21+
submodules: true
2022

2123
- name: Set up Python ${{ matrix.python-version }}
2224
uses: actions/setup-python@v5

‎.github/workflows/miniconda.yml

+4
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ jobs:
2222

2323
steps:
2424
- uses: actions/checkout@v4
25+
with:
26+
submodules: true
2527

2628
- name: Setup Micromamba
2729
uses: mamba-org/setup-micromamba@v1
@@ -53,6 +55,8 @@ jobs:
5355
platform: [x64]
5456
steps:
5557
- uses: actions/checkout@v4
58+
with:
59+
submodules: true
5660

5761
- name: Setup Micromamba
5862
uses: mamba-org/setup-micromamba@v1

‎.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "external/nc_complex"]
2+
path = external/nc_complex
3+
url = [email protected]:PlasmaFAIR/nc-complex.git

‎Changelog

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
version 1.7.0 (not yet released)
22
===============================
3+
* add support for complex numbers via `auto_complex` keyword to `Dataset` (PR #1295)
34
* fix for deprecated Cython `DEF` and `IF` statements using compatibility header
4-
with shims for unavailable functionality
5+
with shims for unavailable functionality (PR #1277)
56

67
version 1.6.5 (tag v1.6.5rel)
78
===============================

‎docs/index.html

+61-11
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<head>
44
<meta charset="utf-8">
55
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
6-
<meta name="generator" content="pdoc 0.10.1.dev7+g3ecfbcf" />
6+
<meta name="generator" content="pdoc 0.10.0" />
77
<title>netCDF4 API documentation</title>
88
<meta name="description" content="Version 1.7.0
99
…" />
@@ -49,8 +49,9 @@ <h2 id="quick-install">Quick Install</h2>
4949
</ul>
5050
<h2 id="developer-install">Developer Install</h2>
5151
<ul>
52-
<li>Clone the
53-
<a href="http://github.com/Unidata/netcdf4-python">github repository</a>.</li>
52+
<li>Clone the <a href="http://github.com/Unidata/netcdf4-python">github repository</a>. Make
53+
sure you either clone recursively, or run <code>git submodule update --init</code> to
54+
ensure all the submodules are also checked out.</li>
5455
<li>Make sure the dependencies are satisfied (Python 3.8 or later,
5556
<a href="http://numpy.scipy.org">numpy</a>,
5657
<a href="http://cython.org">Cython</a>,
@@ -107,6 +108,9 @@ <h1 id="tutorial">Tutorial</h1>
107108
<li><a href="#dealing-with-strings">Dealing with strings</a></li>
108109
<li><a href="#in-memory-diskless-datasets">In-memory (diskless) Datasets</a></li>
109110
</ul>
111+
<p>All of the code in this tutorial is available in <code>examples/tutorial.py</code>, except
112+
the parallel IO example, which is in <code>examples/mpi_example.py</code>.
113+
Unit tests are in the <code>test</code> directory.</p>
110114
<h2 id="creatingopeningclosing-a-netcdf-file">Creating/Opening/Closing a netCDF file</h2>
111115
<p>To create a netCDF file from python, you simply call the <code><a title="netCDF4.Dataset" href="#netCDF4.Dataset">Dataset</a></code>
112116
constructor. This is also the method used to open an existing netCDF
@@ -671,9 +675,9 @@ <h2 id="beyond-homogeneous-arrays-of-a-fixed-type-compound-data-types">Beyond ho
671675
Compound data types
672676
are created from the corresponding numpy data type using the
673677
<code><a title="netCDF4.Dataset.createCompoundType" href="#netCDF4.Dataset.createCompoundType">Dataset.createCompoundType()</a></code> method of a <code><a title="netCDF4.Dataset" href="#netCDF4.Dataset">Dataset</a></code> or <code><a title="netCDF4.Group" href="#netCDF4.Group">Group</a></code> instance.
674-
Since there is no native complex data type in netcdf, compound types are handy
675-
for storing numpy complex arrays.
676-
Here's an example:</p>
678+
Since there is no native complex data type in netcdf (but see
679+
<a href="#support-for-complex-numbers">Support for complex numbers</a>), compound
680+
types are handy for storing numpy complex arrays. Here's an example:</p>
677681
<pre><code class="language-python">&gt;&gt;&gt; f = Dataset(&quot;complex.nc&quot;,&quot;w&quot;)
678682
&gt;&gt;&gt; size = 3 # length of 1-d complex array
679683
&gt;&gt;&gt; # create sample complex data.
@@ -1096,9 +1100,43 @@ <h2 id="in-memory-diskless-datasets">In-memory (diskless) Datasets</h2>
10961100
[0 1 2 3 4]
10971101
&gt;&gt;&gt; nc.close()
10981102
</code></pre>
1099-
<p>All of the code in this tutorial is available in <code>examples/tutorial.py</code>, except
1100-
the parallel IO example, which is in <code>examples/mpi_example.py</code>.
1101-
Unit tests are in the <code>test</code> directory.</p>
1103+
<h2 id="support-for-complex-numbers">Support for complex numbers</h2>
1104+
<p>Although there is no native support for complex numbers in netCDF, there are
1105+
some common conventions for storing them. Two of the most common are to either
1106+
use a compound datatype for the real and imaginary components, or a separate
1107+
dimension. <code><a title="netCDF4" href="#netCDF4">netCDF4</a></code> supports reading several of these conventions, as well as
1108+
writing using one of two conventions (depending on file format). This support
1109+
for complex numbers is enabled by setting <code>auto_complex=True</code> when opening a
1110+
<code><a title="netCDF4.Dataset" href="#netCDF4.Dataset">Dataset</a></code>:</p>
1111+
<pre><code class="language-python">&gt;&gt;&gt; complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j, 1 + 1j, 0.25 + 0.75j])
1112+
&gt;&gt;&gt; with netCDF4.Dataset(&quot;complex.nc&quot;, &quot;w&quot;, auto_complex=True) as nc:
1113+
... nc.createDimension(&quot;x&quot;, size=len(complex_array))
1114+
... var = nc.createVariable(&quot;data&quot;, &quot;c16&quot;, (&quot;x&quot;,))
1115+
... var[:] = complex_array
1116+
... print(var)
1117+
&lt;class 'netCDF4._netCDF4.Variable'&gt;
1118+
compound data(x)
1119+
compound data type: complex128
1120+
unlimited dimensions:
1121+
current shape = (5,)
1122+
</code></pre>
1123+
<p>When reading files using <code>auto_complex=True</code>, <code><a title="netCDF4" href="#netCDF4">netCDF4</a></code> will interpret variables
1124+
stored using the following conventions as complex numbers:</p>
1125+
<ul>
1126+
<li>compound datatypes with two <code>float</code> or <code>double</code> members who names begin with
1127+
<code>r</code> and <code>i</code> (case insensitive)</li>
1128+
<li>a dimension of length 2 named <code>complex</code> or <code>ri</code></li>
1129+
</ul>
1130+
<p>When writing files using <code>auto_complex=True</code>, <code><a title="netCDF4" href="#netCDF4">netCDF4</a></code> will use:</p>
1131+
<ul>
1132+
<li>a compound datatype named <code>_PFNC_DOUBLE_COMPLEX_TYPE</code> (or <code>*FLOAT*</code> as
1133+
appropriate) with members <code>r</code> and <code>i</code> for netCDF4 formats;</li>
1134+
<li>or a dimension of length 2 named <code>_pfnc_complex</code> for netCDF3 or classic
1135+
formats.</li>
1136+
</ul>
1137+
<p>Support for complex numbers is handled via the
1138+
<a href="https://github.com/PlasmaFAIR/nc-complex"><code>nc-complex</code></a> library. See there for
1139+
further details.</p>
11021140
<p><strong>contact</strong>: Jeffrey Whitaker <a href="&#109;&#97;&#105;&#108;&#116;&#111;&#58;&#106;&#101;&#102;&#102;&#114;&#101;&#121;&#46;&#115;&#46;&#119;&#104;&#105;&#116;&#97;&#107;&#101;&#114;&#64;&#110;&#111;&#97;&#97;&#46;&#103;&#111;&#118;">&#106;&#101;&#102;&#102;&#114;&#101;&#121;&#46;&#115;&#46;&#119;&#104;&#105;&#116;&#97;&#107;&#101;&#114;&#64;&#110;&#111;&#97;&#97;&#46;&#103;&#111;&#118;</a></p>
11031141
<p><strong>copyright</strong>: 2008 by Jeffrey Whitaker.</p>
11041142
<p><strong>license</strong>: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:</p>
@@ -1553,7 +1591,8 @@ <h3>Instance variables</h3>
15531591
Ignored if <code>parallel=False</code>.</p>
15541592
<p><strong><code>info</code></strong>: MPI_Info object for parallel access. Default <code>None</code>, which
15551593
means MPI_INFO_NULL will be used.
1556-
Ignored if <code>parallel=False</code>.</p></div>
1594+
Ignored if <code>parallel=False</code>.</p>
1595+
<p><strong><code>auto_complex</code></strong>: if <code>True</code>, then automatically convert complex number types</p></div>
15571596
<h3>Subclasses</h3>
15581597
<ul class="hlist">
15591598
<li>netCDF4._netCDF4.Group</li>
@@ -1582,6 +1621,10 @@ <h3>Static methods</h3>
15821621
</dl>
15831622
<h3>Instance variables</h3>
15841623
<dl>
1624+
<dt id="netCDF4.Dataset.auto_complex"><code class="name">var <span class="ident">auto_complex</span></code></dt>
1625+
<dd>
1626+
<div class="desc"><p>Return an attribute of instance, which is of type owner.</p></div>
1627+
</dd>
15851628
<dt id="netCDF4.Dataset.cmptypes"><code class="name">var <span class="ident">cmptypes</span></code></dt>
15861629
<dd>
15871630
<div class="desc"><p>Return an attribute of instance, which is of type owner.</p></div>
@@ -2627,6 +2670,10 @@ <h3>Instance variables</h3>
26272670
<dd>
26282671
<div class="desc"><p>Return an attribute of instance, which is of type owner.</p></div>
26292672
</dd>
2673+
<dt id="netCDF4.Variable.auto_complex"><code class="name">var <span class="ident">auto_complex</span></code></dt>
2674+
<dd>
2675+
<div class="desc"><p>Return an attribute of instance, which is of type owner.</p></div>
2676+
</dd>
26302677
<dt id="netCDF4.Variable.chartostring"><code class="name">var <span class="ident">chartostring</span></code></dt>
26312678
<dd>
26322679
<div class="desc"><p>Return an attribute of instance, which is of type owner.</p></div>
@@ -3027,6 +3074,7 @@ <h1>Index</h1>
30273074
<li><a href="#parallel-io">Parallel IO</a></li>
30283075
<li><a href="#dealing-with-strings">Dealing with strings</a></li>
30293076
<li><a href="#in-memory-diskless-datasets">In-memory (diskless) Datasets</a></li>
3077+
<li><a href="#support-for-complex-numbers">Support for complex numbers</a></li>
30303078
</ul>
30313079
</li>
30323080
</ul>
@@ -3060,6 +3108,7 @@ <h4><code><a title="netCDF4.CompoundType" href="#netCDF4.CompoundType">CompoundT
30603108
<li>
30613109
<h4><code><a title="netCDF4.Dataset" href="#netCDF4.Dataset">Dataset</a></code></h4>
30623110
<ul class="">
3111+
<li><code><a title="netCDF4.Dataset.auto_complex" href="#netCDF4.Dataset.auto_complex">auto_complex</a></code></li>
30633112
<li><code><a title="netCDF4.Dataset.close" href="#netCDF4.Dataset.close">close</a></code></li>
30643113
<li><code><a title="netCDF4.Dataset.cmptypes" href="#netCDF4.Dataset.cmptypes">cmptypes</a></code></li>
30653114
<li><code><a title="netCDF4.Dataset.createCompoundType" href="#netCDF4.Dataset.createCompoundType">createCompoundType</a></code></li>
@@ -3156,6 +3205,7 @@ <h4><code><a title="netCDF4.Variable" href="#netCDF4.Variable">Variable</a></cod
31563205
<ul class="">
31573206
<li><code><a title="netCDF4.Variable.always_mask" href="#netCDF4.Variable.always_mask">always_mask</a></code></li>
31583207
<li><code><a title="netCDF4.Variable.assignValue" href="#netCDF4.Variable.assignValue">assignValue</a></code></li>
3208+
<li><code><a title="netCDF4.Variable.auto_complex" href="#netCDF4.Variable.auto_complex">auto_complex</a></code></li>
31593209
<li><code><a title="netCDF4.Variable.chartostring" href="#netCDF4.Variable.chartostring">chartostring</a></code></li>
31603210
<li><code><a title="netCDF4.Variable.chunking" href="#netCDF4.Variable.chunking">chunking</a></code></li>
31613211
<li><code><a title="netCDF4.Variable.datatype" href="#netCDF4.Variable.datatype">datatype</a></code></li>
@@ -3198,7 +3248,7 @@ <h4><code><a title="netCDF4.Variable" href="#netCDF4.Variable">Variable</a></cod
31983248
</nav>
31993249
</main>
32003250
<footer id="footer">
3201-
<p>Generated by <a href="https://pdoc3.github.io/pdoc" title="pdoc: Python API documentation generator"><cite>pdoc</cite> 0.10.1.dev7+g3ecfbcf</a>.</p>
3251+
<p>Generated by <a href="https://pdoc3.github.io/pdoc" title="pdoc: Python API documentation generator"><cite>pdoc</cite> 0.10.0</a>.</p>
32023252
</footer>
32033253
</body>
32043254
</html>

‎examples/complex_numbers.py

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
import netCDF4
2+
import numpy as np
3+
4+
complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j, 1 + 1j, 0.25 + 0.75j], dtype="c16")
5+
np_dt = np.dtype([("r", np.float64), ("i", np.float64)])
6+
complex_struct_array = np.array(
7+
[(r, i) for r, i in zip(complex_array.real, complex_array.imag)],
8+
dtype=np_dt,
9+
)
10+
11+
print("\n**********")
12+
print("Reading a file that uses a dimension for complex numbers")
13+
filename = "complex_numbers_as_dimension.nc"
14+
15+
with netCDF4.Dataset(filename, "w") as f:
16+
f.createDimension("x", size=len(complex_array))
17+
f.createDimension("complex", size=2)
18+
c_ri = f.createVariable("data_dim", np.float64, ("x", "complex"))
19+
as_dim_array = np.vstack((complex_array.real, complex_array.imag)).T
20+
c_ri[:] = as_dim_array
21+
22+
with netCDF4.Dataset(filename, "r", auto_complex=True) as f:
23+
print(f["data_dim"])
24+
25+
26+
print("\n**********")
27+
print("Reading a file that uses a compound datatype for complex numbers")
28+
filename = "complex_numbers_as_datatype.nc"
29+
30+
with netCDF4.Dataset(filename, "w") as f:
31+
f.createDimension("x", size=len(complex_array))
32+
nc_dt = f.createCompoundType(np_dt, "nc_complex")
33+
breakpoint()
34+
35+
c_struct = f.createVariable("data_struct", nc_dt, ("x",))
36+
c_struct[:] = complex_struct_array
37+
38+
with netCDF4.Dataset(filename, "r", auto_complex=True) as f:
39+
print(f["data_struct"])
40+
41+
print("\n**********")
42+
print("Writing complex numbers to a file")
43+
filename = "writing_complex_numbers.nc"
44+
with netCDF4.Dataset(filename, "w", auto_complex=True) as f:
45+
f.createDimension("x", size=len(complex_array))
46+
c_var = f.createVariable("data", np.complex128, ("x",))
47+
c_var[:] = complex_array
48+
print(c_var)
49+
50+
with netCDF4.Dataset(filename, "r", auto_complex=True) as f:
51+
print(f["data"])

‎examples/tutorial.py

+8
Original file line numberDiff line numberDiff line change
@@ -355,3 +355,11 @@ def walktree(top):
355355
print(nc)
356356
print(nc['v'][:])
357357
nc.close()
358+
359+
# Write complex numbers to file
360+
complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j, 1 + 1j, 0.25 + 0.75j])
361+
with Dataset("complex.nc", "w", auto_complex=True) as nc:
362+
nc.createDimension("x", size=len(complex_array))
363+
var = nc.createVariable("data", "c16", ("x",))
364+
var[:] = complex_array
365+
print(var)

‎external/nc_complex

Submodule nc_complex added at 37310ed

‎include/netCDF4.pxi

+29
Original file line numberDiff line numberDiff line change
@@ -465,3 +465,32 @@ cdef extern from "netcdf-compat.h":
465465
NC_MPIIO
466466
NC_MPIPOSIX
467467
NC_PNETCDF
468+
469+
470+
# Declarations for handling complex numbers
471+
cdef extern from "nc_complex/nc_complex.h":
472+
bint pfnc_var_is_complex(int ncid, int varid) nogil
473+
bint pfnc_var_is_complex_type(int ncid, int varid) nogil
474+
475+
int pfnc_get_complex_dim(int ncid, int* nc_dim) nogil
476+
int pfnc_inq_var_complex_base_type(int ncid, int varid, int* nc_typeid) nogil
477+
478+
int pfnc_inq_varndims (int ncid, int varid, int *ndimsp) nogil
479+
int pfnc_inq_vardimid (int ncid, int varid, int *dimidsp) nogil
480+
481+
int pfnc_def_var(int ncid, char *name, nc_type xtype, int ndims,
482+
int *dimidsp, int *varidp) nogil
483+
484+
int pfnc_get_vars(int ncid, int varid, size_t *startp,
485+
size_t *countp, ptrdiff_t *stridep,
486+
void *ip) nogil
487+
488+
int pfnc_put_vars(int ncid, int varid, size_t *startp,
489+
size_t *countp, ptrdiff_t *stridep,
490+
void *op) nogil
491+
492+
cdef enum:
493+
PFNC_DOUBLE_COMPLEX
494+
PFNC_DOUBLE_COMPLEX_DIM
495+
PFNC_FLOAT_COMPLEX
496+
PFNC_FLOAT_COMPLEX_DIM

‎setup.py

+15-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import os, sys, subprocess, glob
22
import os.path as osp
3+
import pathlib
34
import shutil
45
import configparser
56
from setuptools import setup, Extension
@@ -411,12 +412,24 @@ def _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs):
411412
osp.join("include", "parallel_support_imports.pxi")
412413
)
413414

415+
nc_complex_dir = pathlib.Path("./external/nc_complex")
416+
source_files = [
417+
netcdf4_src_pyx,
418+
str(nc_complex_dir / "src/nc_complex.c"),
419+
]
420+
include_dirs = inc_dirs + [
421+
"include",
422+
str(nc_complex_dir / "include"),
423+
str(nc_complex_dir / "include/generated_fallbacks"),
424+
]
425+
DEFINE_MACROS += [("NC_COMPLEX_NO_EXPORT", "1")]
426+
414427
ext_modules = [Extension("netCDF4._netCDF4",
415-
[netcdf4_src_pyx],
428+
source_files,
416429
define_macros=DEFINE_MACROS,
417430
libraries=libs,
418431
library_dirs=lib_dirs,
419-
include_dirs=inc_dirs + ['include'],
432+
include_dirs=include_dirs,
420433
runtime_library_dirs=runtime_lib_dirs)]
421434
# set language_level directive to 3
422435
for e in ext_modules:

0 commit comments

Comments
 (0)