wall-mounted-cube¶
English | Japanese
« Home · « External flow and wind engineering
Summary: flow around a cube mounted on the floor of a channel (ERCOFTAC case 41 / SIG15 6.3 (OpenFOAM case in OpenCAE V&V, directory case06.2), the Martinuzzi & Tropea experiment, Re_H = 40,000), solved with the D3Q27 cumulant LBM-LES. The grid is refined H/16, H/32, H/48 and compared with the LDA mean velocity and Reynolds stresses. The H/48 run (1.23 × 10⁸ cells) ran on 8 GH200 of Miyabi-G, which also gave its speed and scaling. Interim summary (2026-10-27), progress of 2026-10-29 below.
Progress (2026-10-29)¶
- Wall-model change of 2026-10-28 does not touch this case. The D3Q27 wall model now takes the tangential velocity relative to the wall (moving floor and wheels of the DrivAer runs). The cube has no moving wall, and the population checksums are bitwise the same before and after the change on GB200: H/32 (20 + 400 steps) and H/48 (20 + 200 steps), stdpar and the CUDA path (first version, new layout, FP32 storage, FP32 collide). All results on this page stay valid.
- Multi-node CUDA on Miyabi-G (1 GH200 per node, one MPI rank per node, CUDA path built with
-cudaon the compute node, source of 2026-10-28). The runs of 2026-10-28 had failed at start (the binary had not been built); rerun on 2026-10-29. s/step including the force reports; the FP64 checksums equal the stdpar ones (ec36499638244575H/48,d33bf4bdf3983bd4H/32), and OVERLAP=3 does not change the result.
| H/48 (24 H span), s/step | 2 nodes | 4 nodes | 8 nodes |
|---|---|---|---|
| stdpar, reduced device halo (2026-10-27) | – | – | 0.02704 |
CUDA LBM_CUDA=3 (FP64) |
0.01511 | 0.00903 | 0.00574 |
CUDA FP64 + OVERLAP=3 |
0.01455 | 0.00813 | 0.00503 |
CUDA LBM_FP32=2 (FP32 collide) |
0.01259 | 0.00779 | 0.00503 |
CUDA FP32 collide + OVERLAP=3 |
0.01200 | 0.00683 | 0.00427 |
Between nodes the overlap of the interior kernels with MPI pays (H/48, 8 nodes: 12-15 %), unlike within one node. H/48 with FP32 collide and overlap on 8 GH200 runs at 0.00427 s/step (29 GLUPS), 6.3x the stdpar run. H/32 (7 H span) is too small for 8 nodes: 0.00059 s/step against 0.00080 on 4.
One node (2026-10-29): H/48 does not run on one GH200 of Miyabi-G (killed by the 100 GiB job memory limit during set-up). H/32 on one GH200: 0.00253 s/step in FP64 (6.6x the stdpar 0.0166) and 0.00206 s/step with FP32 collide (8.1x), checksums as on several nodes. OVERLAP needs more than one rank (HALO_MEM is legacy on one rank). - Near-wall turbulence ahead of the cube (open item below): not started; the first step is still the u_rms measurement on x-planes.
Set-up¶
The cube (edge H) stands on the floor of a closed channel 2H high. The domain runs from 10H upstream of the cube front face to 12H downstream (x/H = -10 to 12). Coordinates as in the experiment: x streamwise, y height above the floor, z spanwise.

| item | H/16 | H/32 | H/48 |
|---|---|---|---|
| grid spacing | H/16 | H/32 | H/48 |
| cells | 1.26 M | 10.1 M | 117 M (123 M with the side-wall solid bands) |
| span | 7H, free-slip sides | 7H, free-slip sides | 24H (as the experiment), no-slip side walls + wall model |
| blockage | 7.1 % | 7.1 % | 2.1 % (as the experiment) |

The grid around the cube (side view, x/H = -1 to 2): uniform Cartesian, the cube is solid cells; 16, 32 and 48 cells over the cube height.
Common settings:
| item | value |
|---|---|
| lattice | D3Q27, cumulant collision, uniform grid (no refinement) |
| LES | Smagorinsky (C_s = 0.10) |
| walls | Spalding-law wall model on floor, top and cube (and the side walls at H/48). Walls are staircase bounce-back on cell faces; no mesh generation |
| inflow | the measured fully developed channel flow at x/H = -18 (LDA mean velocity and Reynolds stresses) with digital-filter synthetic turbulence (length scales L_x = 1.0H, L_y = L_z = 0.3H); no precursor, no recycling |
| outflow | constant pressure |
Results (mean field)¶
H/16 and H/32 ran on 1 GPU, H/48 on 8 GPUs. H/48 averages over 118 H/U_b (5,671 samples) after a 45 H/U_b transient.

Mean velocity U/U_b on the centre plane (z = 0). Circles: experiment (LDA).

u_rms/U_b at the same stations.
| quantity | experiment | H/16 | H/32 | H/48 |
|---|---|---|---|---|
| wake reattachment length X_R/H (first cell / y = 0.02H) | 1.62 (y = 0.02H) | 2.72 / 2.72 | 1.54 / 1.52 | 1.37 / 1.35 |
| front: most upstream reversed flow x/H (first cell) | -1.04 (oil flow) | -0.83 | -1.10 | -1.14 |
| roof: U/U_b at x/H = 0.5 (first cell) | -0.56 (y = 1.02H) | -0.04 | -0.13 | -0.14 |
| first cell centre above the floor y/H | 0.02 (lowest LDA point) | 0.0312 | 0.0156 | 0.0104 |
| mean abs. error U/U_b (all stations) | – | 0.173 | 0.104 | 0.088 |
| mean abs. error u'u'/U_b² | – | 0.0165 | 0.0097 | 0.0080 |
| mean abs. error v'v'/U_b² | – | 0.0161 | 0.0063 | 0.0061 |
| mean abs. error u'v'/U_b² | – | 0.0064 | 0.0036 | 0.0028 |

Running mean of the reattachment length X_R/H (first cell) against the averaging time T U_b/H; dotted: experiment (1.62). Over the last 50 H/U_b it changes by 0.007H at H/32 (1.530 → 1.537) and 0.009H at H/48 (1.383 → 1.374): X_R is converged in the averaging time. H/16 stays at 2.6-2.7 even after 300 H/U_b.
- H/16 to H/32 removes most of the error; H/48 lowers the mean-velocity and stress errors further.
- X_R shortens 2.72 → 1.54 → 1.37; H/48 is 0.25H shorter than the experiment (1.62). Runs that change only the span (H/32 at 24H, H/48 at 7H, several inflow seeds) show a span (blockage) effect on X_R within the seed scatter: the shortening is a grid effect. H/64 (24H, FP32 collide) gives X_R/H = 1.371, within the scatter of H/48 (1.359 ± 0.038): X_R is converged in the grid.
| X_R/H (first cell) | 7H span | 24H span |
|---|---|---|
| H/32 | 1.516 ± 0.025 (5 runs) | 1.489 ± 0.001 (2 runs) |
| H/48 | 1.343 ± 0.023 (2 runs) | 1.359 ± 0.038 (5 runs) |
| H/64 | – | 1.371 (1 run) |
| - The back-flow over the roof stays weak (-0.14 against -0.56). | ||
| - Ahead of the cube (x/H = -3, -1.5) the near-wall turbulence is below the experiment at H/32 and about 50 % above it at H/48. We attribute this to the synthetic-turbulence length scale (0.3H), resolved by 14 cells at H/48, which then decays less; this is not yet verified (see "Open items and remedies"). |
Instantaneous field (H/48)¶
The instantaneous field of the H/48 run's last checkpoint (t = 163 H/U_b).

|U|/U_b on the centre plane z = 0 (x/H = -3 to 12).

Q (H/U_b)² = 10

Q (H/U_b)² = 30
Q-criterion iso-surfaces (Q = ½(|Ω|² − |S|²) by central differences; cells touching a wall left out). The vortices upstream come from the synthetic inflow turbulence, those high up from the boundary layer of the top wall at 2H. At Q (H/U_b)² = 30 the horseshoe vortex wrapping around the cube is visible.
Animations (H/48)¶
The run continued from the last checkpoint (t = 163 H/U_b) and wrote the velocity in x/H = -3 to 11, ±3.5H across, every 48 steps: 160 frames (t = 163.2 to 171.1 H/U_b, a loop of about 10 s).

|U|/U_b on the centre plane z = 0 (x/H = -3 to 11).

Iso-surface Q (H/U_b)^2 = 30 (Q computed as for the stills above, every second cell). Flow from left to right.
Examples in the literature (Q and other iso-surfaces)¶
| reference | conditions | iso-surfaces |
|---|---|---|
| Krajnović & Davidson (2002), AIAA J. 40(5), 927–936 (doi:10.2514/2.1729) | the same ERCOFTAC case 41 (Re = U_b H/ν = 40,000, channel height 2H, span 7H); LES with dynamic one-equation SGS models on coarse grids (16-40 points along a cube edge) | instantaneous Q = 5 (Fig. 9): formation and breakdown of the lateral vortex on the cube side ("ear of a tea cup"); normalisation not stated |
| Nigro et al. (2005), Mecánica Computacional 24, 1299 (paper page) | ERCOFTAC case 41 (Re = U_b H/ν = 40,000, unit cube, span ±3.5); finite-element LES (Smagorinsky, wall law), about 1.3 M tetrahedra, inflow from a periodic-channel precursor | instantaneous Q = 0.35 (Fig. 21: horseshoe, hairpin and lateral vortices); time-averaged Q > 0 shown over the ranges [0.1, 3.0] and [0.2, 0.5] (Fig. 22); normalisation not stated |
| Diaz-Daniel, Laizet & Vassilicos (2017), Int. J. Heat Fluid Flow 68, 269–280 (doi:10.1016/j.ijheatfluidflow.2017.09.015, author version PDF) | DNS, cube in a laminar or turbulent boundary layer, Re_H = U∞H/ν = 500-3,000 | laminar boundary layer, their Fig. 3: Q H²/U∞² = 0.1 (Re_H = 500, 600, 750), 0.2 (1,100), 0.28 (1,700, 3,000); down to 0.003 at Re_H = 500 to show weak unsteadiness. Coloured by streamwise velocity. Hairpin vortices shed periodically from the cube top |
| Schröder et al. (2020), Exp. Fluids 61, 189 (doi, open access) | experiment (time-resolved PIV, 3D Shake-The-Box) and LBM (ProLB), cube in a boundary layer, Re_H = 2,000-8,000 | vorticity-magnitude iso-surfaces, not Q (a laminar-inflow example: |ω| = 170 s⁻¹) |
- The closest to this page are the first two (LES of the same case 41), with Q = 5 and 0.35. The Q (H/U_b)² = 10 and 30 here are larger; a finer grid resolves smaller, more intense vortices, so the level that gives a similar picture generally rises (48 cells per cube edge at H/48 here, 16-40 points in Krajnović & Davidson).
- The Q threshold depends strongly on the Reynolds number, the grid resolution and the velocity used to normalise it (free-stream U∞ above, bulk U_b on this page). The Q (H/U_b)² = 10 and 30 of this page come from an LES at Re_H = 40,000 and cannot be compared directly with the DNS values above (Re_H ≤ 3,000); each study picks the level at which the structures can be told apart.
Speed and scaling (Miyabi-G, GH200)¶
One GH200 per node, one MPI rank per GPU. D3Q27, FP64.
| case | GPUs | s/step | MLUPS | efficiency T₁/(N T_N) |
|---|---|---|---|---|
| H/48 (1.23 × 10⁸ cells) | 1 | 0.1953 | 631 | 100 % |
| 2 | 0.0997 | 1,237 | 98 % | |
| 4 | 0.0542 | 2,273 | 90 % | |
| 8 | 0.0289 | 4,266 | 84 % | |
| H/32 (1.01 × 10⁷ cells) | 1 | 0.0166 | 607 | 100 % |
| 2 | 0.0088 | 1,151 | 95 % | |
| 4 | 0.0051 | 1,990 | 82 % | |
| 8 | 0.0032 | 3,188 | 66 % |
| 1-GPU base | 2-GPU base |
|---|---|
![]() |
![]() |
- The kernels scale almost ideally from 1 to 8 GPUs; the loss of efficiency is the halo exchange (pack, MPI, unpack).
- An improved exchange (only the populations the receiver reads, no wall-ghost exchange, the interior collide and local copies overlapped with MPI) gives at 8 GPUs 0.0263 s/step for H/48 (4,686 MLUPS, 93 %) and 0.0025 s/step for H/32 (4,040 MLUPS, 83 %), bitwise equal to the original. Its 2- and 4-GPU runs are pending; the plots and the table above show the original exchange.
- Weak scaling with 6.2 × 10⁷ cells per GPU (1 → 2 GPUs): 100 %.
- The OpenCAE Symposium 2026 talk gave the speed-up of the H/48 run in the table above (1.23 × 10⁸ cells including the solid bands) as 1.96/3.60/6.76 times on 2/4/8 GPUs from 1 GPU (parallel efficiency 98.0/90.0/84.5 %).
- The H/48 physics run (with FP64 statistics) ran at 0.0277 s/step on 8 GPUs (about 4,450 MLUPS): 156,600 steps in about 1 h 15 min (about 10 node-hours).
Speed and scaling (Flow2, GB200)¶
Flow2 Type II (Nagoya University) has 4 GB200 per node. The three halo exchanges are compared on 1, 2 and 4 GPUs of one node; one MPI rank per GPU, D3Q27, FP64.
| case | GPUs | original s/step (efficiency) | reduced exchange | reduced + overlap |
|---|---|---|---|---|
| H/48 (1.23 × 10⁸ cells) | 1 | 0.1664 (741 MLUPS) | – | – |
| 2 | 0.0835 (99.7 %) | 0.0837 (99.5 %) | 0.0837 (99.4 %) | |
| 4 | 0.0444 (93.8 %) | 0.0440 (94.6 %) | 0.0433 (96.2 %) | |
| H/32 (1.01 × 10⁷ cells) | 1 | 0.01443 (699 MLUPS) | – | – |
| 2 | 0.00747 (96.6 %) | 0.00738 (97.8 %) | 0.00751 (96.1 %) | |
| 4 | 0.00407 (88.6 %) | 0.00387 (93.2 %) | 0.00402 (89.7 %) |
- "Reduced exchange" sends only the populations the receiver reads and skips the wall-ghost exchange; "reduced + overlap" also overlaps the interior collide and local copies with MPI.
- On one GPU the GB200 is about 17 % faster than the GH200 of Miyabi-G (H/48: 0.166 vs 0.195 s/step).
- With the original exchange, 4 GPUs reach 94 % (H/48) and 89 % (H/32), above Miyabi-G (90 %, 82 %, inter-node). The 4 GPUs share one node, so the H/48 halo exchange takes 3.4 ms (2.0 ms with overlap) against 6.3 ms on Miyabi-G, and the improved exchange gains only 1-2 points.
- The population checksums are the same for all three exchanges and equal to those on Miyabi-G (GH200): bitwise identical results across the two machines.
- Two nodes (8 GPUs): pending (Flow2 Type II multi-node jobs have not run yet).
Speed and scaling (Aquarius, A100)¶
Aquarius of Wisteria/BDEC-01 (University of Tokyo; 8 A100-SXM4-40GB per node). One MPI rank per GPU, NVHPC 25.9 with its HPC-X (CUDA-aware), stdpar build. Each row is the fastest of the three halo exchanges.
| case | GPUs (nodes) | s/step | MLUPS | efficiency |
|---|---|---|---|---|
| H/32 | 1 | 0.0305 | 331 | 100 % |
| 2 | 0.0156 | 647 | 98 % | |
| 4 | 0.0082 | 1,226 | 93 % | |
| 8 (1) | 0.0045 | 2,240 | 85 % | |
| 16 (2) | 0.0027 | 3,767 | 71 % | |
| 32 (4) | 0.0019 | 5,423 | 51 % | |
| H/48 | 4 | 0.0993 | 1,241 | 100 % (4-GPU base) |
| 8 (1) | 0.0536 | 2,302 | 95 % | |
| 16 (2) | 0.0285 | 4,322 | 89 % | |
| 32 (4) | 0.0156 | 7,912 | 80 % |
- One A100 is about 0.45-0.5x a GH200 or GB200. H/48 does not fit on one or two 40 GB GPUs.
- On two nodes the reduced exchange pays: H/48 reaches 89 % against 74 % with the original exchange. Overlap adds less than 2 %.
- Checksums equal those on Miyabi-G and Flow2 (four nodes included). On four nodes (32 GPUs) the efficiency is 51 % for H/32 and 80 % for H/48. Eight nodes were not measured.
How large on one node (Flow2, 4 GB200)¶
The 24H span kept, the grid refined; reduced + overlap exchange, 100 steps.
| grid | cells | s/step | MLUPS | peak host memory per rank |
|---|---|---|---|---|
| H/48 | 1.23 × 10⁸ | 0.0433 | 2,849 | – |
| H/64 | 2.83 × 10⁸ | 0.1001 | 2,823 | 68-70 GB |
| H/72 | 4.01 × 10⁸ | 0.1422 | 2,823 | 94-98 GB |
| H/80 | 5.50 × 10⁸ | 0.1902 | 2,890 | 127-130 GB |
| H/88 | 7.31 × 10⁸ | 0.2546 | 2,869 | 167-171 GB |
- The speed stays at about 710 MLUPS per GPU. Host memory is about 930 B per owned cell, so with the job limit (864 GB) one node reaches about H/88.
CUDA path¶
The collide and the stream fused into one CUDA kernel, with the equal-level copy replaced by direct reads from the neighbour blocks; then the populations in an interior-only layout (16-cell rows on 128-byte lines) and the floor/ceiling cells in their own kernel (LBM_CUDA=3). The arithmetic order is that of the stdpar path, and the population checksums are bitwise equal to stdpar on 1-8 GPUs of GB200 and A100. s/step (speed-up against stdpar in brackets).
| GB200 | stdpar | CUDA (first version) | CUDA (new layout) | new layout + FP32 storage | new layout + FP32 collide |
|---|---|---|---|---|---|
| H/48, 1 GPU | 0.1664 | 0.0292 (5.7x) | 0.0222 (7.5x) | 0.0228 | 0.0178 (9.4x) |
| H/48, 4 GPUs | 0.0433 | 0.00846 (5.1x) | 0.00656 (6.6x) | 0.00671 | 0.00545 (7.9x) |
| H/32, 1 GPU | 0.01442 | 0.00271 (5.3x) | 0.00195 (7.4x) | 0.00201 | 0.00156 (9.2x) |
| H/32, 4 GPUs | 0.00404 | 0.00092 (4.4x) | 0.00069 (5.9x) | 0.00071 | 0.00059 (6.8x) |
| A100 | stdpar | CUDA (first version) | CUDA (new layout) | new layout + FP32 collide |
|---|---|---|---|---|
| H/32, 1 GPU | 0.0305 | 0.00693 | 0.00516 (5.9x) | 0.00402 |
| H/32, 4 GPUs | 0.00823 | 0.00222 | 0.00176 (4.7x) | 0.00149 |
| H/32, 8 GPUs | 0.00450 | 0.00148 | 0.00118 (3.8x) | 0.00101 |
| H/48, 4 GPUs | 0.1004 | 0.0289 | 0.0244 (4.1x) | 0.0208 |
| H/48, 8 GPUs | 0.0538 | 0.0194 | 0.0165 (3.3x) | 0.0145 |

Speed-up of the CUDA path against stdpar on the same GPUs (H/48, s/step including the force reports).

Strong scaling of H/48 on Miyabi-G (one GH200 per node, 2026-10-29): time per step (left, dotted lines ideal) and throughput (right). The CUDA path was measured from 2 nodes: on one node H/48 exceeds the job memory limit of Miyabi-G (100 GiB, the GPU memory included) during set-up and is killed; 8 GH200 reach 29 GLUPS with FP32 collide and OVERLAP=3, 6.8x the stdpar run with the original halo exchange.

Speed-up of the CUDA path against stdpar on Miyabi-G (one GH200 per node, the same node count; s/step including the force reports; stdpar with the original halo exchange, CUDA of 2026-10-29). With FP32 collide and OVERLAP=3, H/48 runs 8.3x / 7.9x / 6.8x faster on 2 / 4 / 8 nodes; the speed-up falls as the node count grows (the cause is not yet measured). H/48 does not fit on one node (job memory limit).
- FP32 storage keeps the populations as FP32 in the form f − wᵢ and computes in FP64; FP32 collide also collides in FP32. Statistics and checkpoints stay FP64.
- Asynchronous loads (cp.async) and two streams were tried and did not help.
- Kernel-only time (without the host-side force reports): H/48 on one GB200 0.01925 s/step in FP64 (8.5x stdpar) and 0.01485 s/step with FP32 collide (11.0x); on four 0.00510 / 0.00399 s/step (8.4x / 10.7x).
- The kernel is bound by latency, not by memory bandwidth or FP64 throughput. No fma contraction (only 6-8 % faster) and MPI overlap (no gain within a node) are not the default.
FP32 check¶
H/32 physics runs (45 H/U_b transient + 105 H/U_b averaging), five inflow seeds per variant; ± is the standard deviation over the seeds.
| variant | X_R/H (first cell) | x_F/H | MAE U | MAE u'v' |
|---|---|---|---|---|
| FP64 | 1.516 ± 0.025 | −1.111 ± 0.017 | 0.1032 | 0.0039 |
| FP32 storage | 1.511 ± 0.017 | −1.111 ± 0.018 | 0.1029 | 0.0038 |
| FP32 collide | 1.499 ± 0.029 | −1.111 ± 0.018 | 0.1033 | 0.0039 |
| H/48 FP64 (45 + 60 H/U_b) | 1.359 ± 0.038 | – | 0.0875 | 0.0029 |
| H/48 FP32 collide | 1.368 ± 0.058 | – | 0.0871 | 0.0030 |
- The FP32-collide X_R is 0.017 below FP64, about one standard error of the difference: no significant shift (the 0.05-0.07 seen in single runs was seed scatter). The error metrics agree to the third digit.
- The fastest validated configuration is the new layout with FP32 collide (9.4x stdpar at H/48 on one GB200). At H/48 (45 H/U_b transient + 60 H/U_b averaging, five seeds each) the FP32-collide differences are also within the seed scatter. In physics mode on four GB200, H/48 runs at 0.0052 s/step in FP64 (24 GLUPS) and 0.0040 s/step with FP32 collide (30 GLUPS).
- H/48 with FP32 collide over 45 + 118 H/U_b agrees with FP64 within the tolerances (X_R/H 1.390 vs 1.374, MAE U 0.0863 vs 0.088), so the new layout with FP32 collide (
LBM_CUDA=3 LBM_FP32=2) is the production physics mode; the FP64 new-layout mode stays as the bitwise check against stdpar. The H/64 physics run (2.8 × 10⁸ cells) takes 0.0088 s/step on four GB200: 192,000 steps in one hour.
Open items and remedies¶
Near-wall turbulence ahead of the cube¶
Now: u_rms/U_b at y = 0.15H is below. The H/48 peak sits at y = 0.14-0.18H, the measured one at the lowest point (0.02H). The same excess shows near the top wall (y = 2H).
| x/H | experiment | H/32 | H/48 |
|---|---|---|---|
| -3 | 0.094 | 0.084 | 0.135 |
| -1.5 | 0.103 | 0.094 | 0.160 |
How the inflow is made: the mean velocity and Reynolds stresses measured at x/H = -18, set to 0 at the walls, are imposed at the inlet (x/H = -10) with digital-filter fluctuations. The filter length scales are constant over the height (L_y = L_z = 0.3H, time correlation L_x = 1.0H). There is no precursor or recycling, so the result depends on whether the synthetic turbulence develops into wall turbulence over the 7H from the inlet to x/H = -3.
Candidate causes (none verified yet):
- The length scale stays 0.3H near the walls, larger than wall eddies (which shrink in proportion to the wall distance). At H/48 it is resolved by 14 cells and is carried downstream without decaying (10 cells at H/32, which decays sooner).
- The amplitude at the inlet itself: the measured stresses are imposed through the Cholesky factor, possibly including parts the grid cannot resolve.
- Over-predicted near-wall u_rms of wall-modelled LES: with a wall model the streamwise fluctuation can come out too large in the first few cells.
Remedies (in this order):
- Measure to separate them: u_rms profiles on x/H = -9.5, -8, -6, -3 planes, to separate the inlet value from the decay or growth downstream. Too large already at the inlet → 2; growing downstream → 3; right at the inlet but decaying too slowly → 1. The drivers write only y/z planes now, so an x-plane output is needed.
- Wall-distance length scales: reduce L_y, L_z near the walls to about min(0.3H, κ d) (d the wall distance, κ ≈ 0.41), set in units of H, not cells. Check at H/32 and H/48 whether u_rms at x/H = -3 and -1.5 moves to the experiment.
- Precursor inflow: run a periodic channel at the same grid spacing and feed its planes to the inlet. The most reliable, but needs time-series plane input/output.
- Inlet position: moving the inlet further upstream (-10H → -15H, say) gives the synthetic turbulence more length to develop. Compare with 2 if measurement 1 shows a too-slow decay.
Others¶
Separate the grid effect from the span (blockage) effect on X_R: done (span effect within the seed scatter, the shortening is a grid effect, converged at H/64). Why X_R is shorter than measured remains open.- Spanwise-periodic and lateral no-slip boundary options (H/48 builds its side walls from bands of solid cells).
- The host-side set-up (topology, partition) limits one job to about 1.5-2 × 10⁸ cells.
Build and run on Miyabi-G¶
Checked on 2026-10-29 with the public main (scripts in scripts/case041/miyabi/) and nvhpc 25.9 + HPC-X 25.9. Miyabi-G has one GH200 per node: run one MPI rank per node. A job's memory limit (100 GiB) also counts what is placed in the GPU memory.
1. Source and inflow data¶
Login node.
export GROUP=<group> # project group, e.g. gz00 (`id -Gn` lists yours)
export CASE_DIR=/work/$GROUP/$USER/case041 # work directory of the case
git clone https://gitlab.com/masaz/openrheo.git && cd openrheo
scripts/case041/fetch_data.sh # ERCOFTAC case 41 data (not in the repository)
mkdir -p $CASE_DIR/bin && python3 scripts/case041/make_inflow.py $CASE_DIR/inflow_x01.dat
/work is divided by project group, so give the group explicitly: on Miyabi id -gn returns the personal group, not the project group.
2. Build¶
On a compute node, about 2 min. scripts/case041/miyabi/build.pbs builds the single-GPU and multi-GPU drivers; CUDA=1 adds the multi-GPU driver with the CUDA path (-cuda), orheo-forest-cube-channel-d3q27-mpi-cuda:
qsub -W group_list=$GROUP -v REPO=$PWD,BIN_DIR=$CASE_DIR/bin,CUDA=1 scripts/case041/miyabi/build.pbs
The build log is the job output c41-build.o<job id> in the directory where qsub was run (the speed check writes c41-speed.o<job id>). The compiler prints about 30 warnings there (lambda captures, kernel launches); they are expected. The last line is === build done on success, === build FAILED or === CUDA build FAILED otherwise; grep -v -i warning c41-build.o* | grep -E "===|error" shows only those.
3. Speed and checksum check¶
scripts/case041/miyabi/run_speed.pbs; the node count is select, one rank per node. It runs H/48 (24 H span, 20 + 200 steps) and H/32 (7 H channel, 20 + 400 steps) in five modes: stdpar with the original and with the reduced halo exchange plus overlap, CUDA FP64, CUDA FP64 + OVERLAP=3, CUDA FP32 collide + OVERLAP=3; other modes with MODES (see the header of the script). On 2 nodes the whole job takes about 4 min.
qsub -W group_list=$GROUP -l select=2 -v CASE_DIR=$CASE_DIR scripts/case041/miyabi/run_speed.pbs
The job writes c41-speed.o<job id> in the directory where qsub was run; on Miyabi the file grows while the job runs, so tail -f c41-speed.o* follows it. At the end the job prints a summary, one line per run, with the checksum compared with the reference (OK, NG, or no reference for configurations other than the default). On 2 nodes:
=== summary (2 nodes)
H/48 stdpar rc=0 s/step=0.10205 ec36499638244575 OK
H/48 stdpar-ovl rc=0 s/step=0.10010 ec36499638244575 OK
H/48 c3 rc=0 s/step=0.01508 ec36499638244575 OK
H/48 c3ov rc=0 s/step=0.01449 ec36499638244575 OK
H/48 c3f32ov rc=0 s/step=0.01200 b65b5a0b67e14afd OK
H/32 stdpar rc=0 s/step=0.00896 d33bf4bdf3983bd4 OK
H/32 stdpar-ovl rc=0 s/step=0.00878 d33bf4bdf3983bd4 OK
H/32 c3 rc=0 s/step=0.00153 d33bf4bdf3983bd4 OK
H/32 c3ov rc=0 s/step=0.00150 d33bf4bdf3983bd4 OK
H/32 c3f32ov rc=0 s/step=0.00126 3c28d3a5ebaef9a1 OK
=== all runs OK
s/step is the time per step including the force reports (the TIMING line); TIMING_KERNELS above the summary is the kernels alone. Reference checksums:
| grid | FP64 (stdpar, stdpar-ovl, c3, c3ov) | FP32 storage (LBM_FP32=1) |
FP32 collide (LBM_FP32=2) |
|---|---|---|---|
| H/48 (20 + 200 steps) | ec36499638244575 |
6a28de6f5831daf7 |
b65b5a0b67e14afd |
| H/32 (20 + 400 steps) | d33bf4bdf3983bd4 |
d5ae5874c7720640 |
3c28d3a5ebaef9a1 |
The FP64 modes give the stdpar checksum, the bitwise check. Positional arguments of the driver: cells per H, steps, U_b in lattice units (0.05), Re_H (40000), Smagorinsky constant (0.10). On more than one node, the settings reach the other ranks through OMPI_MCA_mca_base_env_list (set in the script).
4. Physics runs¶
Runs with statistics, profiles and restarts: scripts/case041/miyabi/run_mpi_phys.pbs (several GPUs) and run_single.pbs (one GPU) run the series in pieces that fit the queue's walltime; each piece writes a checkpoint and the next piece is submitted after it ends (no dependency chains). The usage is in the header of each script, e.g.
qsub -W group_list=$GROUP -l select=8 -v CASE_DIR=$CASE_DIR,TAG=h48,NH=48,TRANS=43200,TOTAL=400000 scripts/case041/miyabi/run_mpi_phys.pbs
These scripts use the stdpar binary (build.pbs); the 2026-10-27 physics runs with the CUDA path were made on Flow2 GB200.
5. Plots and comparison with the experiment¶
Login node; checked on 2026-10-29 with the H/48 physics run of 3 pieces.
Python with numpy and matplotlib: the system python3 of Miyabi has an old numpy and no matplotlib, so make a virtual environment once (or pip3 install --user --upgrade numpy if ~/.local already has matplotlib; that affects every use of the system python3):
module load python/3.11.15
python3 -m venv /work/$GROUP/$USER/venv-c41
/work/$GROUP/$USER/venv-c41/bin/pip install numpy matplotlib
PY=/work/$GROUP/$USER/venv-c41/bin/python
Profiles and QoI against the LDA data (all pieces of the run, in order; about 20 s):
RUN=$(ls -d $CASE_DIR/runs/h48/p* | paste -sd,)
$PY scripts/case041/compare_case041.py --figs $CASE_DIR/figs --run H48=$RUN --json $CASE_DIR/figs/qoi.json
It writes 14 figures (centre-plane and off-centre U, V, W and Reynolds stresses at the SIG15 stations, near-wall U, centre-plane streamlines, QoI convergence) and qoi.json (X_R/H, front and roof reversed flow, mean absolute errors). The compact set of the summary figures on this page:
LAST=$(ls -d $CASE_DIR/runs/h48/p* | tail -n 1) # the last piece (p01, p02, ... sort in order)
$PY scripts/case041/make_summary.py --out $CASE_DIR/summary --run H48=$LAST \
--qoi $CASE_DIR/figs/qoi.json --qoi-label H48=H48
firefox $CASE_DIR/summary/*.png # view (needs ssh -X; or copy the files to a PC)
writes centreline_{U,urms,vrms}.{png,pdf} (x/H = -1.5, 1, 2, 3), qoi_table.{md,csv} and xr_convergence.{png,pdf}. Several runs can be compared with more --run LABEL=....
Instantaneous field in ParaView, in three steps (wait for each job to end before the next step):
# 1. velocity box from the newest checkpoint (the same node count and settings as the run;
# AVG_FROM = TRANS of the run once the averaging has started)
CK=$(tail -1 $CASE_DIR/runs/h48/state | awk '{print $3}')
qsub -W group_list=$GROUP -l select=8 -v CASE_DIR=$CASE_DIR,NH=48,CK=$CK,AVG_FROM=43200,OUT=$CASE_DIR/runs/h48/inst scripts/case041/miyabi/inst_dump.pbs
After that job has ended:
# 2. one VTK file (login node)
$PY scripts/case041/inst_to_vti.py $CASE_DIR/runs/h48/inst/inst $CASE_DIR/inst.vti --stride 2
# 3. images with ParaView pvbatch (login node; Xvfb gives it a virtual display)
mkdir -p $CASE_DIR/render
module load paraview/5.13.1
xvfb-run -a -s "-screen 0 2400x1400x24" pvbatch scripts/case041/render_inst.py $CASE_DIR/inst.vti $CASE_DIR/render 30 100
firefox $CASE_DIR/render/*.png # view (needs ssh -X; or copy the files to a PC)
The dump of H/48 is 1.7 GB (8 ranks); --stride 2 gives a 100 MB .vti in a few seconds (without it the full 1056 x 384 x 96 box). The .vti has U, |U|, Q (in (U_b/H)^2) and the solid mask; render_inst.py draws |U| on the centre plane and one Q iso-surface per level (Q (H/U_b)^2 = 30, 100). The images umag_y0.png, q_30.png, q_100.png go to $CASE_DIR/render in about 10 s. The ParaView module of Miyabi-G (5.13.1) needs a display, hence xvfb-run; with ssh -X (XQuartz on a Mac) its GUI paraview opens the .vti directly. Alternatives: scripts/case041/miyabi/render_inst.pbs runs the same rendering as a Miyabi-C (x86_64) job with the ParaView for Linux x86_64 from paraview.org (PARAVIEW = its directory), or copy the .vti to a PC and open it in ParaView there.
Validation¶
| Tier | T2 — single-point comparison |
| QoI | X_R/H, U/U_b, Reynolds stresses |
| Reference | ERCOFTAC case 41 (Martinuzzi & Tropea, LDA), X_R/H = 1.62 |
| Gate | — no gate |
| Note | Grid study H/16-H/64 against the experiment, X_R converged in the grid at H/64; not T0, which needs an analytic solution |
Source¶
The drivers, the halo/compute overlap and the CUDA path are in the public main (merged on 2026-10-29; scripts/case041/ holds the data, inflow, comparison and Miyabi-G job scripts, including miyabi/run_speed.pbs for the speed and checksum check over the halo, overlap and CUDA modes). Experimental data: ERCOFTAC Classic Collection, case 41.

