Fiducial Parameters

global.data

nDim     = 2                            ! number of dimensions for this problem (1-3)
GmX      = 42,420,1                     ! Base grid resolution [x,y,z]
MaxLevel = 7                            ! Maximum level for this simulation (0 is fixed grid)

GxBounds = 0d0,0d0,0d0,6d0,60d0,0d0     ! Problem boundaries in computational units,format:
                                        ! (xlower, ylower, zlower, xupper, yupper, zupper)
                                        ! For 2D problems, set zlower and zupper to 0.d0.

Gmthbc   = 5,4,1,1,1,1                  ! Sets the physical boundary conditions at the edge of the problem domain
                                        ! format:  (x1, y1, z1, x2, y2, z2)
                                        ! 1-Extrapolated,
                                        ! 2-Periodic,
                                        ! 3-ReflectingWall (Field lines do not penetrate)
                                        ! 4-Reflect_BParallel (Field lines held normal
                                        ! 5-Reflect_Cylindrical (Like reflect wall, but also changes sign of phi
                                        !                        components of velocity and magnetic fields)
                                        ! [1,1,1,1,1,1]

!=============================================================================================
! Parameters Related to Time
!=============================================================================================
start_time        = 0d0                 ! The initial time in computational units [0d0]. Do NOT change this on restarts.
final_time        = .2866183232d0                       ! The final time in computational units.
final_frame       = 10                  ! The final frame [10]
lRestart          = F                   ! restart flag [F]
lPostProcess      = F                   ! Post Process (preempts lrestart) [F]
restart_frame     = 47                  ! The frame from which restarts should begin.  Ignored unless lRestart or lPostProcess = .true.
initial_maxspeed  = 1000.d0             ! A lower bound on the initial maximum wave speed in computational units.
cfl_vars          = .8, .4, .1          ! Maximum allowed CFL before restarting step, desired CFL to run at, relaxation parameter.  Smaller numbers give more gradual relaxation [1, .3, .5]


!=============================================================================================
! Parameters Related to Performance/Algorithms
!=============================================================================================
lSkipProfile            = T                             ! Flag for skipping profiling [T] - should set this to false for good performance on production runs.
lUseOriginalNewSubGrids = F                             ! T for Old Subgrid Algo, F for New Subgrid Algo. [F]
MinimumGridPoints       = 2                             ! The minimum number of cells a grid can have along one dimension [2] (ignored unless lUseOriginalNewSubGrids=T).
iThreaded               = -1                                    ! -1 is non-threaded, 0 is pseudo-threaded, 1 is threaded [0].  If you are using threaded you must compile with pThreadFlag=2
LevelBalance            = 0d0,0d0                       ! Try to balance load on every level? [0d0,0d0]

/

!============================================================================================
!Level-specific variables (All LevelData lists should be initialized from 0 to MaxLevel).
!============================================================================================

&LevelData
qTolerance              = .10,.25,.40,1d30,1d30,1d30,1d30,1d30          ! field variable tolerances on each level
DesiredFillRatios       = .70,.70,.70,.70,.70,.70,.70,.70       ! Determines how much of the new subgrids must be filled with refined cells (also ignored unless lUseOriginalNewSubGrids=T)
buffers                 = 4, 4, 4, 4, 4, 4, 4, 4                ! ErrFlag buffering
/

!============================================================================================
!Level-specific variables (All LevelData lists should be initialized from 0 to MaxLevel).
!============================================================================================

&PoissonData
solver                  = 1             ! Which solver to use for linear system solve? StructPCG=1, StructGMRes=2 [1]
tolerance               = 1e-6          ! Solver convergence tolerance [1e-6]
printlevel              = 0             ! Sets print level for hypre solver.  Increasing this will make solvers more verbose. [0]
MaxIters                = 1000          ! Maximum number of iterations to attempt convergence.  [1000]
hverbosity              = 0             ! Anything > 0 will cause linear systems to be outputted as matrix, rhs, lhs files. [0]
mthbc                   = 2 2 2 2 2 2   ! What type of boundary conditions to use for gravitational potential
                                        ! format:  (x1, y1, z1, x2, y2, z2)
                                        ! 0-Zero Slope,
                                        ! 1-Reflecting,
                                        ! 2-Periodic,
                                        ! 3-Multipole Expansion,
                                        ! 4-User Specified (User is reponsible for setting Phi at physical boundaries)

physics.data

iEOS            = 1            ! Equation of State [0]-ideal gas, 1-MultiSpecies gas, 3-Van der Waals gas, 4-Isothermal
gamma           = 1.666667d0   ! Adiabatic index used for ideal gas and Van der Waals gas [1.66666666667]
IsoTemp         = 10d0         ! Isothermal temperature for Isothermal EOS
CriticalDens    = 30.693677103 ! maximum density in Van der Waals EOS
AttractionParam = 1.38d6       ! attraction pram in Van der Waals EOS
/
&AbundanceData
HeFrac = 0.085d0
xH     = 0.01d0
xHe    = 0d0
xHeII  = 0d0
ZFrac  = 0.001165d0
Zmu    = 16.48d0
Xmu    = 0d0
xZ     = 4.13127d-4
/

&PhysicsData

!================================================================================
! Field based refinement control
!================================================================================
InterpOpts = 0,0,0,0,0,0,0,0,0,0,0,0,0  ! [0]-constant, 1-minmod, 2-superbee, 3-vanLeer, 4-mc, 5-Parabolic (not conservative), 6-Linear (not limited).  It is recommended to use constant interpolation for any fields in q that correspond to aux fields (iBx, iBy, iBz, etc...)
refineVariableFactor = 1d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0,0d0 ! weight factors for each field used in triggering refinement.  Combined with qtolerance for that level. [0d0]

!================================================================================
! Global source term switches
!================================================================================
lSelfGravity   = .false.    ! Turns on self-gravity if true [.false.]
UniformGravity = 0d0        ! Gravitational acceleration in the -y direction [0d0]
OmegaRot       = 0d0        ! Shifts to a rotating reference frame (enables coriolis and centripetal force terms)
iCylindrical   = 2          ! [0]-No cylindrical geometry, 1-Cylindrical with no angular momentum, 2-Cylindrical with angular momentum
iCooling       = 4          ! Cooling module [NoCool=0, AnalyticCool=1, DMCool=2, IICool=3, ZCool=4]
SrcPrecision   = 1d-3       ! Relative precision desired for source terms.  [1d-3]
lTrackHydrogen = .true.
lTrackHelium   = .true.
ne_min=1d-3
FloorTemp=1d3
!================================================================================
! Explicit solver switches
!================================================================================
lResistive     = .false.     ! Turns on resistivity [.false.]
ResType        = 2           ! Resistivity type. 1-constant, 2-user defined distribution, 3-Spitzer. [1]
resistivity    = 1.2d-1      ! resistivity value [0d0]
lConductive    = .false.     ! Turns on mhd heat conduction [.false.]
ConType        = 1           ! Conduction type. 1-constant, 2-Spitzer. [1]
conductivity   = 8d-2        ! conductivity value [0d0]
lThermalFluxLimiter= .false. ! Turns on saturation heat flux. (only works for ConType==2). [.false.]
lIsoDiff       = .false.     ! Turns on isotropic heat conduction. Allows isotropicity to be in the range (0,1). [.false.]
lViscous       = .false.     ! Turns on viscosity (2D only) [.false.]
VisType        = 1           ! viscosity type. 1-classic, 2-Braginskii [1]
bviscosity     = 0d0         ! viscosity value [0d0]

!================================================================================
! Density Protection Options
!================================================================================
lRestartOnDensityProtections = .false.  ! Do density protections trigger restarts?  [.false.]
iDensityProtect  = 2                    ! 0-Set to MinDensity, 1-Set to minimum nearby density, [2]-Average nearby densities
iMomentumProtect = 2                    ! 0-Conserve momentum, 1-Set to zero, [2]-Average nearby velocities
MinDensity       = 1d-4                ! Minimum computational density before protection is triggered [1d-10]

!================================================================================
! Pressure Protection Options
!================================================================================
lRestartOnPressureProtections = .false. ! Do pressure protections trigger restarts? [.false.]
iPressureProtect = 2                    ! 0-Set to MinTemp, 1-Set to minimim nearby pressure, [2]-Set to average nearby pressure, 3-Set to minimum nearby temperature, 4-Set to average nearby temperature
MinTemp          = 1d-10                ! [1d-10] minimum allowed temperature for the system in Kelvin before protection is triggered

!================================================================================
! Other Protection Options
!================================================================================
lRestartOnHypreMaxIters = T             ! Restart steps if hypre does not converge

!================================================================================
! Description of various scaling parameters
! Define one of each of the following: [nScale/rScale], [TempScale/pScale], and set the other to 0d0.
!================================================================================
nScale          =       1d2,              ! number density scale parameter [particles/cc]
rScale          =       0,                ! density scale [g/cc]
TempScale       =       1d4,                ! temperature scale parameter [Kelvin]
pScale          =       0d0,              ! pressure scale [dynes/cm^2]
lScale          =       5e15,       ! length scale parameter [cm] (AU=1.49598e13, pc=3.08568025e18, R_sun=6.955e10

!================================================================================
! MHD related section
!================================================================================
lMHD             =  .false.      ! Magnetic Fields present? [.false.]
lCheckDivergence =  .false.      ! Turn on divergence checking [.false.]

solver.data

        iScheme = 1     ! Sweep = [1]
        iSolver = 0     ! [0]-Default HLL Solver, 1-iExactRS, 2-HLLC, 3-HLL, 4-HLLC_ISO, 6-HLLD, 8-HLLD_ISO
/
&SweepData
  InterpOrder = 2          ! Interpolation order 1-Piecewise constant, 2-Piecewise linear, [3]-Piecewise Parabolic
  lChar_Limiters = .true.  ! Limit using characteristic variables instead of primitive? [.true.]
  lCautious=.true.         ! Protect quantities throughout hyperbolic step [.true.]
  ViscCD = 0               ! [0] = NO_VISCOSITY, 1 = ROE_VISCOSITY, 2 = H_VISCOSITY
  iTracer = 0              ! [0] = NONLAGRANGIAN, 1 = LAGRANGIAN
  lApplyDiffusion=.true.   ! Apply additional diffusion at converging flows [.true.]
  DIFF_ALPHA = .1          ! Diffusion coefficient [.1]
  lApplyLOF = .false.      ! Apply local oscillation filter (reduces carbuncles) [.false.]
  LOF_ALPHA = .075         ! Strength of local oscillation filter. [.075]
  lCTU=.true.              ! Use corner transport upwind?  [.true.]
  lLimiter=.true.          ! Use additional slope limiters [.true.]

problem.data

namb = 1d2           ! ambient density [cm^-3]
tamb = 1d4           ! ambient temperature [K]
njet = 5d2           ! jet density [cm^-3]
vjet = 200d5          ! initial jet velocity [cm/s]
vrange = 50d5         ! (+/-) range of jet velocities [cm/s]
Rjet = 1d0           ! jet radius [cu]
Rm = 0.6d0          ! radius at which maximum B field is reached
Betam = 1d0        ! plasma beta sets the maximum B field
Bgeom = 1           ! jet magnetic field geometry (1 = toroidal, 2 = helical)
vtype = 1            ! type of veloicty pulse (0 = constant, 1 = sinusoidal, 2 = random)
freq = 10d0             ! frequency of velocity pulses in units of 1/final_time
                     ! so if sinusoidal, this is how many sine waves the velocity will go through
                     ! if random, this is how many times the the velocity will change to a new value
theta = 0d0            ! angle of precession of the injection velocity [deg]
precfreq = 30d0        ! frequency of the precession in units of 1/final_time
seed = 1             ! number used to seed the random number generator (allows runs to be repeatable
                     ! and ensures that each processor uses the same random velocity)
simtype = 0          ! 0 = single jet, 1 = colliding jets (fields rotating in same direction)
                     ! 2 = colliding jets (fields rotating in opposite direction)
N = 1               ! number of sub-sampling iterations
CoolingRes = 10      ! number of cells per cooling length (using a velocity of 50 km/s)

Key Parameters

Resolution: 42X420 + 7 AMR

cfl_vars = .8, .4, .1

qTolerance            = .10,.25,.40,1d30,1d30,1d30,1d30,1d30

iThreaded             = -1

InterpOpts            = 0,0,0,0,0,0,0,0,0,0,0,0,0

buffers                 = 4, 4, 4, 4, 4, 4, 4, 4

MinDensity       = 1d-4[

MinTemp          = 1d-10

iDensityProtect  = 2

iMomentumProtect = 2

iPressureProtect = 2

	Fiducial	5 AMR	3 AMR	Buffer increase	Density Refinement
% root steps restarted	3.66%
Time to frame 1	1min
Last frame	15
Time for last frame	30 mins
Est. time to completion	6 hours
Number of processors	256
Location
Other deviations from Fiducial
Last Frame / movie link

Resolution Tests

The idea here is a bit different than adjusting data file parameters to see what works best. Here, I am simply lowering the number of levels while keeping the number of processors constant. This is to see how having a larger root grid might help performance when running on many processors.

There is density gradient refinement to level 3, and cooling time threshold refinement to level 7 (based on fiducial run). qTolerance changes for each run to keep this consistent. For example, the level 6 run has

qTolerance = .25, .40, 1d30, 1d30, ...

so that there is the same refinement criteria at the same effective resolution. For level 5…

qTolerance = .40, 1d30, 1d30, 1d30, ...

The level 4 and level 3 runs have the same qTolerance…

qTolerance = 1d30, 1d30, 1d30, 1d30, ...

	Fiducial	6 AMR	5 AMR	4 AMR	3 AMR
% root steps restarted
Time to frame 1
Last frame
Time for last frame
Est. time to completion
Number of processors	1008	1008	1008	1008	1008
Location	kraken	kraken	kraken	kraken	kraken
Other deviations from Fiducial	N/A	qTolerance (see above)	qTolerance (see above)	qTolerance (see above)	qTolerance (see above)
Last Frame / movie link