agmip / translator-dssat Goto Github PK

We need to refactor this plugin to find more efficient ways of making the translators palpable to people. Good start though, but because DSSAT typically used ICASA variable names, we shouldn't need to do this much work.

We need to be able to handle an input of atmospheric CO2. This should be handled in the environmental modifications section of FileX. If a date is not supplied, use the start of simulation date.

Input translator 2-char -> 3-char
Output translator 3-char -> 2-char

for both variables.

(Maybe already handled - please verify)

The output for simulation control lose order since changed from LinkedHashMap to HashMap. It will be revised by changing to use key to get value instead of order.

crid is currently located in the dssat_sequence section of the JSON. This needs to actually be under a planting event under management\events.

The date variable should be in 'yyDDD' format rather than 'yyyyDDD'. If use 'yyyyDDD', DSSAT model still can successfully run but the simulation result would be wrong.

The format of Date variable in weather file is in page 42 of this doc.
Reference: https://github.com/DSSAT/dssat-csm-resources/blob/main/Technical%20Documentation/DSSATv35/DSSAT%20vol2.pdf

The SDAT from the simulation controls should be picked up from DSSAT input files and stored with the data.

It is caused by a recent update of dssat weather translator. The new handling to sort daily data by year has a bug to divide the data correctly.

Since the DSSAT translator receives the entire experiment list in the same order it was generated in, when it processes in parallel the order gets shuffled. Please try to retain the order while still processing quickly.

This could be handled in the following manner:

1. Received the map
2. Parse the map, extracting (and maybe setting) only the name of the experiment into an ArrayList.
3. Process the map.
4. Loop through the arraylist when generating the DSSAT Batch to set the order.

Just an idea.

It should be used for output XFile when it is valid.

The XFileOutput should be changing the Simulation Controls CO2 from M to W. This stopped working even though the code is in there.

This bug may NOT be in this package, but it is the place to start looking.

Out of memory issues:

When running large numbers of simulations, the system runs out of memory. I will work on profiling this and finding the choke points.

ITEMS OF NOTE:

• Close all open files
• Do not create new maps, use references to the current map being passed.
• Set things to null when you are finished with them.

It looks like the observed section does not match what was discussed in the Gainesville meeting. The new structure should be as follows:

"observed":{
    "hwam":"value",
    ...
    "time_series":[{
        "var":"value",
        ...
    }]
}

All tests should refer to paths within the src/test/resources directory and be loaded by classloader interface. This will help all the tests to pass without hardcoding the paths.

After applying the latest patch, I find that we still have only one treatment per experiment (vs 30) even with the double underscore being applied. These need to be multiple treatments under the same experiment.

When DSSAT creates a multi-treatment XFile, we need to make sure that if TRT_NAME is supplied, we use that.

We just need to make sure that Organic Matter is being used within the translator.

Need to transition from LinkedHashMap to HashMap for memory performance issues. This should be a simple LinkedHashMap to HashMap search and replace across the entire project.

Moved the files from subdirectories per experiment to all files in the base directory of the zip. Need to make sure that the DSSATBatch.v45 file points to the correct files.

Problem: a dataset included "hacom" (harvest component). This was correctly put into a harvest event under management in the json. The DSSAT translator created a harvest event, but the model failed because there was no date associated with the event.

Solution: The translator should create a harvest event and give the date of the observed harvest. If there is no observed harvest date, then don't create the event.

ICASA variable VPRSD = DSSAT variable VPRS

The DSSAT translator is using VPRSD as the weather file header, but should be using VPRS.

When the daily weather is not a full year data in the *.wth file, the input translator might give a wrong sorting result.

For single treatments, use the EXNAME for the treatment name. Right now, we have no way of knowing which experiment is which in the Summary.OUT file because of the problem with the experiment being out of order.

You may have to truncate the name to fit the field.

Data and headers need to be right-justified in the DSSAT weather file. The model parses data by columns based on header locations. Image: file on the left is from Weatherman, file on the right from translator-dssat. We will handle the TAV and AMP variables later, this issue is just about the formatting of the data beneath headers.

This refers to DSSAT experiment GHWA0401.mzx, which is an NxP experiment using the soil analysis section to define initial soil P for each treatment.

When the experiment is converted to aceb using quadui-1.2.1-SNAPSHOT-Beta19, then re-converted to DSSAT, one of the soils is missing from the reconstituted data.

Input translator:
When fallow treatments are encountered, create a planting event with only CRID:"FAL" (no date, etc.)

Possible issues:
How will the DOME handle an event with no date?

Weather station elevation not being picked up from the JSON.

Source data from DSSAT file GHWA0401.MZX contains a soil analysis section with phosphorus and organic matter info. When these are output to the soil file, only the soil analysis data are written. All other soil data are -99.

this could possibly be a problem with the input translator rather than the output translator. The relevant files are here:
https://drive.google.com/file/d/0ByUhvGN3ZVgXLXlTZWhwTmkyN3M/view?usp=sharing

Documentation for the JSON needs to be updated.

The translator needs to check for dssat_cul_id inside the planting event and if not found, return a -99. Currently, it is using the cul_id.

The latitude should be shown in YCRD, and longitude should be shown in XCRD.

When dealing with multiple treatment reductions (such as the Machakos dataset), there is only one field specified when there should be many fields (one per unique combination of weather and soil type).

The ICASA variable SLRGF is not being translated to the DSSAT variable SRGF. See Wheat pilot JSON in the dropbox.

When writing the soil file, it seems like the soil file is being written incorrectly and missing a lot of the soils. Please make sure that the .SOL file has all the soils included in the JSON string. It might be occurring because of the transition from multiple directories to a single directory.

WST_ID needs to allow more than 4-characters in both field and weather data. DSSAT rules need to rename from non-standard to DSSAT-standard names.

The model is picking up the treatment name from the first line of the FILEX, which currently contain comments from the translator.
Solution: move the comments to below the first line.

When CO2Y is provided in the map, the CO2 switch in Simulation controls should be set to "W" so that the CO2 value in the weather file is used.

When no CO2Y value is provided or if the value is negative, then the CO2 switch in the Simulation controls should be set to "M" (default value).

What happens is a name which does not match the DSSAT specifications gets passed to the translator? Such as an experiment with spaces, commas, or longer than 8 characters?

Soil profile names must be 10 characters in DSSAT.

ICASA standards list PLDP as mm
DSSAT uses PLDP = cm

Make sure that the input translator is converting correctly.
Thanks,

create dssbatch.v47 file and run47.bat file.

dssat_sequence should be renamed to dssat_simulation_controls or something similar. Cheryl can comment on this one for the best name.