Performance & Scaling on Other Codes

This page is summarizes the scaling tests used by other codes, as well as their results. We are, of course, much cooler than they are, but we can nonetheless learn from their results.

• \(l\) is a specific level.
• \(g(l)\) is the number of grids on level \(l\).
• \(p\) is the number of processors.

CASTRO

Although CASTRO can run both on pure MPI and an OpenMP-MPI hybrid, this discussion will concern itself with pure MPI.

• Test Specs
  • All tests performed on the jaguarpf machine at OLCF.
  • Weak scaling at 64³ base grid per core.
  • Time for one core to advance one cell: 24.8 microseconds.

• Test runs:
  • Fixed-grid, no gravity
  • Fixed-grid, monopole gravity
  • Fixed-grid, Poisson gravity
  • 1-level AMR, monopole gravity

• Results:
  • Very good scaling from 8 to 64k processors.
  • Pure MPI does not perform as well as OpenMP-MPI hybrid. No stats available for non-Poisson problems.
  • AMR introduced overhead from 5% at 8 procs to 19% at 64k procs.

ENZO

• Performance Test
  • Two tests: Fixed-grid vs. AMR.
  • Track wall-clock time on each time step (broken down by physics), CPU time on each processor.

• Scaling Test:
  • Fixed-grid tests use cell updates/sec/CPU vs. processor count as a metric; AMR only uses time-to-solution.
  • No specs given for AMR scaling tests.

• Fixed-grid Test Specs
  • 4 problem sizes: 256³, 512³, 1024³, 2048³
  • 3 clusters: Lemieux (Compaq DEC Alpha), Mercury (IBM Itanium2), DataStar (IBM Power4).

• Results:
  • Mercury departs from ideality at 16 processors due to slow communications fabric.
  • DataStar, Lemieux become non-ideal at 32 processors.

Orion

Flash

• Tests Run:
  • Strong scaling test in fixed-grid and AMR.

• Test Spec:
  • Run on ASCI Red (Sandia National Laboratory), Pentium II Xeon, 256 MB/node
  • Sedov problem.

References

​Norman et al. 2007 — Enzo parallelization.

​Almgren et al. 2010 — CASTRO parallelization.

​Fryxell et al. 2000 — FLASH parallelization.

​Rendleman et al. 1999 — An additional resource on parallelization of AMR algorithms.