amrex-codes / exaepi Goto Github PK

We should have a an exaepi_used_inputs like warpx_used_inputs that stores the exact parameters used for each run so we'll have it later.

@debog I added OpenMP parallelization to the MFIter loops in AgentContainer.cpp - could you try to look at the OpenMP scaling again when you get a chance?

We should start to run nightly tests on biollante and garuda for a few inputs files.

HIP tests are currently failing on development branch.

We need to output data on the number of people obeying quarantine so it can be used to calibrate SQIR models. We also should have a few policies that can be toggled between in the inputs.

Currently, the max_grid_size parameter is exposed and needs to be set to a reasonable value based on the number of cells and the number of MPI ranks. This should be computed automatically, and max_grid_size should only be used by advanced users.

When the same executable is run multiple times, the answers will be different to a few parts in a thousand - we think the answers should be exactly the same.

I am trying etc/plot.py to extract ExaEpi results. I have update AMReX, ExaEpi, and installed yt from source. But I still get the error:

(pyamrex) [hvandam@lpo-170127 plt00000]$ python ../../../etc/plot.py .
yt : [INFO     ] 2023-12-15 13:37:25,009 Parameters: current_time              = 0.0
yt : [INFO     ] 2023-12-15 13:37:25,009 Parameters: domain_dimensions         = [150 149   1]
yt : [INFO     ] 2023-12-15 13:37:25,009 Parameters: domain_left_edge          = [0. 0. 0.]
yt : [INFO     ] 2023-12-15 13:37:25,009 Parameters: domain_right_edge         = [1. 1. 1.]
Traceback (most recent call last):
  File "/home/hvandam/Documents/AMReX/ExaEpi/examples/test-pbar+1/plt00000/../../../etc/plot.py", line 10, in <module>
    ad = ds.all_data()
         ^^^^^^^^^^^^^
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/data_objects/static_output.py", line 1194, in all_data
    self.index
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/data_objects/static_output.py", line 606, in index
    self._instantiated_index = self._index_class(
                               ^^^^^^^^^^^^^^^^^^
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/frontends/boxlib/data_structures.py", line 926, in __init__
    super().__init__(ds, dataset_type)
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/frontends/boxlib/data_structures.py", line 345, in __init__
    GridIndex.__init__(self, ds, dataset_type)
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/geometry/geometry_handler.py", line 42, in __init__
    self._setup_data_io()
  File "/home/hvandam/pyamrex/lib64/python3.11/site-packages/yt/frontends/boxlib/data_structures.py", line 568, in _setup_data_io
    self.io = io_registry[self.dataset_type](self.dataset)
              ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'boxlib_native'
(pyamrex) [hvandam@lpo-170127 plt00000]$ 

As far as I can see boxlib is much more common that boxlib_native but I don't know enough about the details to be able to judge how similar these keys are.

Hey guys,

I recompiled for the new updates and was able to run on a single process all the way through. I then went to use the four gpus and currently getting some issues. Here is some output.

I'm launching the allocation with

salloc --nodes 1 --qos interactive --time 01:00:00 --constraint gpu --ntasks-per-node 4 --gpus-per-task 1 --account=m3623_g

I then run simply with

srun /path/to/agent inputs &> outputs

Sample output:

Initializing AMReX (24.03-35-gb892132f6144)...
MPI initialized with 4 MPI processes
MPI initialized with thread support level 0
Initializing CUDA...
CUDA initialized with 4 devices.
AMReX (24.03-35-gb892132f6144) initialized
Total pop 6783760
Total workers 3342275
Number of communities: 4967
Setting initial case counts in 2630 disease hubs.
Base domain is: ((0,0) (70,69) (0,0))
Max grid size is: 16
Number of boxes is: 25 over 4 ranks.

Infected / FIPS / Total Infected
731 6001 733
630 6013 1365
477 6041 1843
108 6055 1954
344 6075 2298
331 6081 2633
836 6085 3473
338 6095 3814
237 6097 4053
Simulating day 0
Writing plotfile
Simulating day 1
Simulating day 2
Simulating day 3
Simulating day 4
0::Assertion `mmc[3] == counts[4]' failed, file "/global/homes/k/keilbart/git/ExaEpi/src/main.cpp", line 221 !!!
SIGABRT
See Backtrace.0 file for details
MPICH ERROR [Rank 0] [job id 23529530.1] [Tue Mar 26 14:18:22 2024] [nid001565] - Abort(6) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0
srun: error: nid001565: task 0: Exited with exit code 255
srun: Terminating StepId=23529530.1
slurmstepd: error: *** STEP 23529530.1 ON nid001565 CANCELLED AT 2024-03-26T21:18:23 ***
srun: error: nid001565: tasks 1,3: Terminated
srun: error: nid001565: task 2: Terminated
srun: Force Terminated StepId=23529530.1

Let me know if I need to provide other information.

Getting these warnings when compiling on Lassen with GCC-12.

/usr/workspace/keilbart/Lassen/ExaEpi/build/_deps/amrex-src/Src/Base/AMReX_Math.H:149:15: note: parameter passing for argument of type 'std::pair<float, float>' when C++17 is enabled changed to match C++14 in GCC 10.1
149 | std::pair<float,float> sincos (float x)

Link to source code documentation in the main readthedocs page.