Skip to content

Commit 679597d

Browse files
committed
ENH: runtime HEATMAP/DXT cross-check log
* add a carefully crafted darshan log using latest `darshan` `main` branch that contains "runtime" `HEATMAP` module and DXT tracing, to facilitate the writing of carefully designed tests for internal consistency of data structures (and to facilitate rapid visual inspection as well) * see the `README` for a very detailed description of how the data was produced (this is perhaps a bit too detailed, though I'm mostly doing this so that more folks can generate carefully-crafted testing data using pure Python without waiting for me for example)
1 parent 9eaa543 commit 679597d

File tree

3 files changed

+58
-0
lines changed

3 files changed

+58
-0
lines changed
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
This is an example darshan log that contains both runtime
2+
heatmap module data and DXT tracing, to facilitate both
3+
visual and robust numerical comparison of the two approaches
4+
of generating heatmaps in the new Python summary report.
5+
6+
The log follows the following "graphical" IO pattern
7+
of writing only a single byte of data per rank, for each
8+
of 32 ranks, on staggered/exaggerated delays:
9+
10+
Time Bins -->
11+
Rank
12+
|
13+
|
14+
V
15+
31 x
16+
30 x
17+
29 x
18+
28 x
19+
..
20+
3 x
21+
2 x
22+
1 x
23+
0 x
24+
25+
The example application code used to generate the log
26+
is available here:
27+
https://github.com/tylerjereddy/heatmap_diagonal
28+
29+
In particular, commit hash `4712f54` from that project
30+
was used, and the precise timing and related details
31+
may be obtained from that project. In short, 32 bytes (1 per rank)
32+
were written at minor timing offsets in a Python mpi4py-based
33+
module on a single node of the LANL snow supercomputer
34+
(Intel skylake gold). Only, POSIX, DXT_POSIX, and
35+
(runtime) HEATMAP modules should be present. There is
36+
no *read* activity in the log (just 1 byte written
37+
per rank).
38+
39+
The commit hash of the darshan main branch used to build
40+
the darshan runtime as `b4932a02a`.
41+
42+
Here is the command line that was used on a
43+
snow backend node:
44+
`mpirun -x DXT_ENABLE_IO_TRACE=1 -x DARSHAN_LOGPATH=/yellow/users/treddy/generated_darshan_logs -x DARSHAN_DISABLE_SHARED_REDUCTION=1 -x LD_PRELOAD=/yellow/users/treddy/darshan_install/lib/libdarshan.so -np 32 python heatmap_diagonal.py`
45+
46+
Here are the steps there were followed to build
47+
the `darshan-runtime` on a snow backend:
48+
49+
```
50+
module load gcc/9.3.0
51+
module load openmpi/3.1.6
52+
module load hdf5-parallel/1.10.7
53+
./prepare.sh
54+
cd darshan-runtime/
55+
./configure --prefix=/yellow/users/treddy/darshan_install --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID --enable-hdf5-mod --with-hdf5=/usr/projects/hpcsoft/toss3/snow/hdf5/1.10.7_gcc-9.3.0_openmpi-3.1.6 CC=mpicc
56+
make
57+
make install
58+
```

darshan_logs/runtime_and_dxt_heatmaps_diagonal_write_only/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)