I ran the test simulation on orda10 again overnight, this time using CFL=0.3 and a MaxLevel of 3. This resulted in another bad CFL in the 14th frame:

 14  trun  00: 04: 22: 51.38 tmessage  00: 00: 00:  0.00 tcompute  00: 04: 17: 18.79
 Bad CFL =   97407161193.4217
  Horribly bad CFL. Die.

This morning I ran the simulation again, dialing the MaxLevel back to 2 and it ran fine. Upon upping the CFL to 0.4, however, I got another bad CFL, this time in the 13th frame:

 13  trun  00: 00: 27: 38.65 tmessage  00: 00: 00:  0.00 tcompute  00: 00: 27: 11.64
 Bad CFL =   92333375773.4226
  Horribly bad CFL. Die.

A just-for-the-hell-of-it run at CFL = 0.5, MaxLevel = 1 also executed okay, leading me to suspect that the problem is related mostly to the MaxLevel. A chart of the results so far is as follows:

CFL / Ref. Level	1	2	3
0.3	OK	OK	BAD
0.4	OK	BAD	BAD
0.5	OK

I am currently running a test at CFL = 0.5, MaxLevel = 0.2, which I expect to fail. I'm also going to rerun CFL = 0.3, MaxLevel = 2 just to make sure the first test wasn't a fluke. If all three MaxLevel = 2 tests fail, then I'm willing to declare the MaxLevel the problem.

UPDATE: I finished running the following tests: CFL=0.5, MaxLevel = 0.2, CFL=0.3 (again—it still ran fine), MaxLevel = 0.2, CFL=0.2, MaxLevel = 0.1, and CFL=0.2, MaxLevel = 0.2. The new results chart is below:

CFL / Ref. Level	1	2	3
0.2	OK	BAD
0.3	OK	OK	BAD
0.4	OK	BAD	BAD
0.5	OK	BAD

CFL = 0.3, MaxLevel = 0.2 still confuses me, because it is the only MaxLevel 2 simulation I have been able to* complete so far. Attached is a tar of the .data files for this project (it's essentially the project used in the setup documentation, only with CFL = 0.3 and MaxLevel = 1.

UPDATE TO THE UPDATE: I have successfully run a test for CFL = 0.1, MaxLevel. I'm now going to try running MaxLevel = 1 simulations stepping up the CFL # until I approach 1.0 or the CFL breaks.

FINAL UPDATE FOR THE DAY: I just ran MaxLevel = 1 tests for CFL = {6.0, 7.0, 8.0, 9.0}. The new results chart is below:

CFL / Ref. Level	1	2	3
0.1	OK	BAD
0.2	OK	BAD
0.3	OK	OK	BAD
0.4	OK	BAD	BAD
0.5	OK	BAD
0.6	OK*
0.7	OK*
0.8	OK
0.9	BAD

• The timestep at this point was now big enough that the last frame could not be calculated. I'm guessing, though, that a 29-frame test was probably good enough.

This test failed, but not because of a bad CFL error. The UNPHYSICAL DENSITY PROTECTION warning was popping up, but did not stop execution. The run did, however, slow down considerably. Then around the 18th frame it suddenly crashed and started rapidly writing the same line over and over again:

18  trun  00: 00: 41: 42.98 tmessage  00: 00: 00:  0.00 tcompute  00: 00: 41: 39.59

until the log file was over 1 GB in the space of a couple minutes. Needless to say, I killed the process at that point. I also took the liberty of download the log file off the server, compressing it locally, and deleting the server copy.