Introduction

HYPRE is a general software package for solving linear systems on massively parallel computers. We intend to use HYPRE as our Poisson solver for self gravity.

The Structure of HYPRE

HYPRE's structure (as far as is seen by the user) depends upon the chosen "interface." There are several different interfaces available; for our purposes, the two of interest are the "Structured" interface and the "Semi-Structured" interface. Different solvers may be accessed depending on the interface.

The specifics of how one interfaces with HYPRE depends on what interface is being used.

The Structured Interface

The simplest interface, the Structured Interface is appropriate for non-AMR grids, such as fixed grid or domain-decomposed calculations. If all the grid cells in the domain are the same size and share a common index space, in which there is everywhere a 1:1 correspondence between cells, the structured interface may be used. There is no need to explicitly indicate that two grids lie next to each other; if one grid goes from i=1:15 and another from i=16:21, they are understood to be neighbors along the i=15:16 interface.

The Semi-Structured Interface

The Semi-Structured Interface is appropriate for AMR or otherwise non-uniform grids. (In the case of AMR, there will be a 2:1 correspondence between cells on a fine level and cells on the coarse level.) With this interface, the each grid is assigned to a "part." Grids in a given part must share a common index space. For simplicity, we take each part to correspond to each AMR level. More explicit spatial relationships between grids is required in this interface. Similarly, the weights of the stencil components which overlap coarse/fine interfaces must be of higher order.

*The rest of this page will discuss the structured interface with a domain-decomposed problem. Discussing the semi-structured interface will be required as we get closer to implementing HYPRE fully with AstroBEAR.*

Interacting with HYPRE

There are several steps which must be completed when using HYPRE (in parallel). They are as follows:

1. Initialize MPI.
2. On each processor, create and initialize a "StructGrid," one for every subdomain of the grid owned by the current processor. Here you specify only the extents of each grid. Once all the grid extents have been specified, the StructGrid is "Assembled" (finalized).
3. On each processor, the discretization stencil is defined. In 3D, with the cell center defined as (0,0,0), the standard 7-point stencil is defined as

   stencil= (( 0, 0, 0),   ! center
             (-1, 0, 0),   ! left
             ( 1, 0, 0),   ! right
             ( 0,-1, 0),   ! bottom
             ( 0, 1, 0),   ! top
             ( 0, 0,-1),   ! back
             ( 0, 0, 1))   ! front
   

4. On each processor, create, initialize, and populate a "StructMatrix." The StructMatrix contains the stencil weights for the stencil as defined in the previous step, for each cell in each grid owned by the current processor. For convenience, instead of setting each grid cell separately, a range of indices may be specified for which all the stencil weights are the same. This then requires a single call to "StructMatrixSeetBoxValues" for those cells.
5. On each processor, set the matrix components for the boundary cells. For example, setting the stencil weights to zero at the edges (boundaries) of any grids which are on the edge of the computational domain. Editor's note: What about periodict boundaries? This doesn't matter much for self-gravity, but still.
6. On each processor, once the above two steps are completed we may "Assemble" (finalize) the matrix.
7. On each processor, "StructVectors" for the source vector and the solution vector must be created, initialized, and populated. This is done in a manner similar to the above, now with calls to "StructVectorSetBoxValues." After this is done, the vectors are "Assembled" (finalized).
8. On each processor, set up and use a solver. Parameters for the solve, such as the maximum number of iterations, convergence tolerance, or the printing of solver information, are specified here.
9. On each processor, recover the solution vectors for grids on the current processor. This is done with "StructVectorGetBoxValues" calls analogous to the above. The solution may also be printed directly to an ASCII text file, if requested.
10. On each processor, destroy the HYPRE objects which were created (Grid, Stencil, Matrix, Solution Vector, Source Vector, Solver) as well as anything else which does not need to be in memory anymore (index ranges, lists of grids, etc.).

Implementing HYPRE in AstroBEAR

There are two main roadblocks to putting HYPRE into AstroBEAR:

1. Worker processors do not know anything about the grids they own, the master processor must tell them.
2. Worker processors do not know anything about other processors' grids which neighbor their grids.