hoang-dt / nasa-encoding-framework Goto Github PK

Hi @dbalouek and @Zhang690683220 ,

As discussed earlier, I think one of the immediate goals is to convert a large dataset from NASA to OpenViSUS (or idx2 later).

The LLC4320 dataset is supposedly 4PB uncompressed and can be obtained by the following command (on NAS):

/home4/bcnelson/MITgcm/extract/latest/uncompress4320 -L 1 10368-19584 U

The above command extracts field U, time steps 10368 to 19584 , for a total of 64 time steps (each increases the timestep label by 144, so 19584 = 10368 + 144 * 64).
The full details can be found in data-descriptions.

Once the raw data is extracted, it can be converted to OpenViSUS (see conversion). @aashishpanta0 should know the details of this conversion.

We can also transfer parts of the data from NAS to SCI for easier testing if needed (see data transferring).