AstroBEAR Test Suite

Any living code base requires checks to make sure that its continued development does not introduce any new bugs. AstroBEAR's developers are working to create a comprehensive lists of tests (the test suite) which runs a set of simulations in order to ensure the fundamental behavior of the code is robust and unchanged.

Current testing (new, 6/30/11)

At the moment we do post-processing tests of astroBEAR2.0. Basically, tests consist on comparing the flow variables in a chombo file produced by the tests modules, against a reference chombo file (see below). All test related files and directories are located in modules/tests/. Each test has its own directory which contains:

• the problem's data files (global.data, modules.data, problem.data, profile.data, solver.data, communication.data, io.data and physics.data),
• the problem's module (problem.f90),
• bear2fix.data, bear2fix.data.test and bear2fix.data.img which are BearToFix data files (see below),
• a "ref" directory which contains the reference chombo file.

The reference chombo file. This file, chombo00001.hdf, has been produced with the code and verified by someone in our research group using well documented, and quantitative, analytical or numerical studies. Reference chombos are included with the current distribution of the code.

Running tests

Tests report. Once test runs are finished, BearToFix will be executed. It will do the error computations and produce two png images showing the reference and the current result of the run. This images will be linked to corresponding wiki pages of the tests. BearToFix is executed twice; the 1st time it read bear2fix.data.img to set the colour tables for the test report images, whereas the 2nd time it read bear2fix.data.test to perfom the erroc computation and produce the test report images.

Tests error calculations. These are done in BearToFix via L2norms and LInfinity norms

To use BearToFix by yourself (e.g. in order to do further adjustment of the test report parameters), select operation 11, test 4 (Generic Test) and input the maximum mean error tolerance (see below) allowed for each fluid variable. BearToFix will then produce the test report images and links to their wiki pages.

If the test fails, BearToFix outputs the current errors as well as suggested new tolerances (5% higher then the current errors). The user can then copy and paste these figures into the bear2fix.data.test file.

---

The basic design philosophy is as follows:

• There exist 3 levels of interactivity and completeness when running the tests —
  1. ALL tests may be run via alltests.sh (Est. time: ? min)
  2. All STRICTLY CODE VERIFICATION tests may be run via verifytests.sh (Est. time: ? min)
  3. Individual tests may be run via their own runtest.sh files, located in their respective folders.

• The 3 run scripts are all located in the contrib/astro/tests folder, and are intended to be run in that location after a successful build of the code.
  • Note: Prior to checkins, we should require developers to run verifytests.sh, which performs barebones verification tests.
  • Periodically run alltests.sh on the code with no user interaction & post results to web.

• The tests should run and report results with minimal interaction required. Ideally, running a test and viewing the result should be as easy as executing a single script.
  • This will require scripts which correctly runs the executable on a given platform.
  • The run should be designed to produce data and exit in a repeatable fashion (i.e., not requiring the user to stop the simulation after a certain time).
  • Upon completion, the script needs to invoke bear2fix which, via the input-file option, processes the data and produces a result.

• The exact nature of the result will be different depending on the test. For simple tests, the result may consist of little beyond "pass" or "fail."
• For other tests, or in cases where failure is reported, the result may need to include certain figures, such as comparison figures (comparing the result versus the expected result).
• Some tests will not necessarily produce success/failure results. These "showcase" tests will instead produce images which show the endstate of the simulation. Such images will then be viewed by the user(s) for confirmation. These tests in other words are meant to illustrate whether the code functions as expected in a general sense when running well-understood simulations. Since it is difficult, especially from the beginning, to design tests to cover every possible failure mode of the code, these tests serve as an important indicator that something is going wrong. (They therefore also serve as an important starting point in diagnosing such.)

• For simplicity, all verification tests will run mpibear on the current machine using 2 processors. More scripting will be needed to run larger showcase simulations if they need to run in parallel with 2+ cores in order to finish in a reasonable amount of time.

List of tests

Verification tests

Test folder name	Brief description	Result type	Est. time to run	Test wiki page
01-1D_sod_shock-tube	Runs 1D Sod shock tube with all combinations of integration schemes	P/F with rel. error, text file	60 sec. (single-proc, grass)	TestSuite/SodShockTube
07-Field_Loop_Advection	Advects a loop of magnetic field around the grid with different field strengths and CT methods	P/F with rel. error, text file	30 minutes (2-proc, grass)	TestSuite/FieldLoop2D
??-Radiative shock instability	Two related tests to verify the cooling source term(s)	P/F with rel. error, text file	unknown	TestSuite/RadiativeShocks

Showcase tests

Test folder name	Brief description	Result type	Est. time to run	Test wiki page
06-2D_shocked_clump	Over-dense clump in pressure equilibrium is overrun by a planar shock	Multi-panel PNG image showing fluid variables (n, vx, vy, p, T, clump tracer) at end of simulation	5 min. (single-proc, grass)	To-Do

Still to be tested:

• Robustness of build on multiple architectures (MPICH2, OpenMPI, VAMPICH, BlueGene; Intel, GFortran)

CASTRO tests

• Scalability
  • 64³ weak scaling test
    • No gravity
    • Multipole gravity
    • Poisson gravity

• hydrodynamic solver
  • Sod Shock Tube (Sod, 1978)
  • Double rarefaction (Toro, 1997)
  • Strong shock (Toro, 1997)

• hydrodynamics, geometry
  • 1D Sedov-Taylor blast wave (Sedov, 1959)
  • 2D cylindrical Sedov-Taylor blast wave (Sedov, 1959)
  • 3D Cartesian Sedov-Taylor blast wave, 1D (Sedov, 1959)

• hydrodynamics, gravity
  • Split piecewise linear, Rayleigh-Taylor (Taylor, 1950)
  • Unsplit piecewise linear, Rayleigh-Taylor (Taylor, 1950)
  • Split PPM (old limiter), Rayleigh-Taylor (Taylor, 1950)
  • Unsplit PPM old limiter), Rayleigh-Taylor (Taylor, 1950)
  • Split PPM (new limiter), Rayleigh-Taylor (Taylor, 1950)
  • Unsplit PPM (new limiter), Rayleigh-Taylor (Taylor, 1950)