ncar / adf Goto Github PK

The original NCAR diagnostics package generated a website that contained all of the relevant diagnostic figures. Not only does this greatly improve the organization of the figures and the diagnostics' ease of use, but it also means that many scientists' workflows assume that a website can be generated. Thus this new version of the diagnostics needs the ability to generate a website as well.

However, there needs to be a discussion on the following topics:

1. What will this new website look like?
2. What kind of information will the website contain?
3. How will the information be organized on the website?

before development can begin.

Need to add a new script which plots the zonal average of a given variable, as well as the latitude-dependent difference between the zonal averages for two different model runs (or a model run and observations).

The actual plotting script will be written by @brianpm.

I'm going through the video of the walk-through and there are few things that aren't up to date in the wiki. Thought I'd start this here and others can add what they've noticed needs to be changed.

1. https://github.com/NCAR/ADF/wiki/Contributor's-Guide

• Replace CAM_diagnostics with ADF
• Others?

2. https://github.com/NCAR/ADF/wiki/CAM-diagnostics-package-examples#running-a-cam-vs-cam-comparison-on-casper

• Replace CAM_diagnostics with ADF (Step 5
• Replace config_cam_baseline.yaml with config_cam_baseline_example.yaml
• Others?

Due to the presence of certain libraries (in particular the "pathlib" library), the CAM diagnostics only works with Python version 3.4 (or later).

Ideally, these library calls should be replaced with older methods to allow for a wider range of python versions to be used, including Python version 2.7.

So I noticed that the climo files output by the ADF are per variable? This is different from the all variables into one seasonal file. Just wondering why that was done, as of course people will have scripts that work off of the original all-in files. Sorry if I have missed a switch somewhere that will enable one to do this!
Rich

As was mentioned in the meeting, it would be good to have an example of running this from a Jupyter Notebook

As pointed out by @juliecaron, currently the website directory, which contains all of the files needed for the static website, are stored in the top-level directory specified by cam_diag_plot_loc, which means it can be overwritten if the diagnostics are run multiple times for multiple different model comparisons.

To alleviate this concern, it would be better if the website directory was a subdirectory under the auto-generated X_vs_Y directory, where X is the case of interest and Y is either obs or the name of the baseline case. This will allow the user to run the diagnostics for multiple different cases without having to worry about accidently over-writing the website files for each case.

Many plotting scripts use very similar functions to generate figures, which may results in lots of repeated, and potentially inefficient, code.

To avoid this, it would help to create a CAM diagnostics plotting functions library which can (optionally) be used by all the plotting scripts to perform certain oft-repeated actions, which should hopefully reduce the overall code footprint, and allow for a single location where modifications and/or bug-fixes can be made.

Currently if one want to skip time series generation, but still compute the climo files, then one needs to add the variable:

cam_ts_done: True

To their config (YAML) file. However, the example baseline config file does not contain this option, so for the new user its existence may not be known, and they may end up having a failed run. Thus this option should be added to the example baseline config file, with enough written in the comments to hopefully make its use self-explanatory.

This will ensure that the ADF framework still works as expected even if a user forgot to rebase and/or merge up their fork.

This issue will also be used to ensure the repo name-change didn't impact the PR system.

In order to help in the development and maintenance of the CAM diagnostics package, unit tests (either as official python "unittests" or as "doctests") should be developed such that the diagnostics code can be easily tested, including in an automated CI/CD workflow.

Along with this, a method to properly apply the pylint linter program to the CAM diagnostics would also be beneficial, as it would help catch any possible syntax errors, as well as improve readability and portability. Ideally this method can also be automated in a CI/CD workflow, similar to the unit tests.

As new DASK-enabled tools such as GeoCAT and Intake-ESM are brought into the package, there is a potential to use DASK to significantly speed-up computation.

However, to allow for user control, and to optimize the use of DASK across the entire package, an over-arching DASK framework, possibly in the form of a new ADF class or object, should be implemented.

New feature type

New plot and/or plot enhancement

What is this new feature?

we have lat/lon plots of scalar fields, but i'm adding the vector field option

Assistance required?

Yes, I will need some assistance before making a PR

Extra info

wish me luck!!

The CAM diagnostics package needs an optional --debug flag that can be used to output various messages and values to a debug log file. This will help both users and developers gather info that is potentially helpful when trying to fix various problems, but which is likely unnecessary for a regular diagnostics run.

I know it seems trivial, but I don't think ncar_pylib will run if python is not loaded - so could add module load python explcitly that into the docs somewhere. Non pythoners could fall over this I guess. Just a thought.

For most ADF diagnostic runs there are usually quantities describing the simulations being used (model start and end date, horizontal resolution, available model variables, etc.) that if not provided by the user in the config file may require the opening of at least a few model files, and that can be useful to scripts throughout the ADF.

Thus it would be helpful to create a new ADF class that extracts this info directly from the model case files when not provided by the user, and make it available to any scripts that require it. Given that ecg-tools can basically do all of this already, this new class could serve as an interface for ecg-tools/Intake-ESM (issue #25) and the rest of the ADF.

Diagnostics configuration files would benefit from the ability to reference other variables contained within the YAML file itself. For example, instead of having to do this:

#Name of CAM case (or CAM run name):
cam_case_name: new_cam_run

#Location of CAM climatologies:
cam_climo_loc: /some/path/to/climo/new_cam_run

One could instead do:

#Name of CAM case (or CAM run name):
cam_case_name: new_cam_run

#Location of CAM climatologies:
cam_climo_loc: /some/path/to/climo/$cam_case_name

That way the user would only have to modify a minimal amount of entries if they wanted to do multiple diagnostic runs with the data saved/stored in the same general locations.

Both @juliecaron and @andrewgettelman have received the following error when running the diagnostics with the variable Q (specific humidity):

Q has lev dimension.

Traceback (most recent call last):

  File "./run_diag", line 178, in <module>

    diag.create_plots()

  File "/glade/campaign/cgd/amp/jcaron/python/CAM_diagnostics/lib/CamDiag.py", line 1028, in create_plots

    function_caller(plot_func_name, plot_func_args, func_kwargs=plot_func_kwargs, module_name=plot_func_name+'.py')

  File "/glade/campaign/cgd/amp/jcaron/python/CAM_diagnostics/lib/CamDiag.py", line 175, in function_caller

    return func(*func_args, **func_kwargs)

  File "/glade/campaign/cgd/amp/jcaron/python/CAM_diagnostics/lib/../scripts/plotting/zonal_mean.py", line 183, in zonal_mean

    oseasons[s], ops_season, ohya, ohyb)

  File "/glade/campaign/cgd/amp/jcaron/python/CAM_diagnostics/lib/plotting_functions.py", line 330, in plot_zonal_mean_and_save

    diffnorm = normfunc(vmin=np.min(diff), vcenter=0.0, vmax=np.max(diff))

  File "/glade/u/apps/dav/opt/python/3.7.9/gnu/9.1.0/pkg-library/20201220/lib/python3.7/site-packages/matplotlib/colors.py", line 1136, in __init__

    raise ValueError('vmin, vcenter, and vmax must be in '

ValueError: vmin, vcenter, and vmax must be in ascending order

This is very likely a bug, and thus should be fixed accordingly.

Currently the CamDiag.py class has a large number of class methods, some of which are relatively complex, while others are basically repeating the same code as others. Thus it would likely help both the readability of the code and the ability to expand and automatically test the code if additional modularization and simplification occurred.

Specifically, it would help if some of these methods were inherited from other, simpler diagnostic classes, while other methods that have repeated codes should be combined into a single routine.

In order to get intake-esm in here, we need to add these packages to the environment

In example_namelist.yaml, under diag_cam_baseline_climo,
there's a key called calc_cam_climo, but in CamDiag.py it looks for the name calc_bl_cam_climo.

I think that the error is in CamDiag.py, but I'm not 100% sure.

As currently written, the create_time_series method requires the ncrcat NetCDF Operator, which is not a standard OS command, nor is it python-based. Thus its use makes porting the CAM diagnostics more difficult, as it requires additional external libraries.

On the other hand, the ncrcat algorithm is very efficient, and any replacement function written in python will likely be much slower. Thus using it will help maintain a quicker run-time for the diagnostics package, at least for certain configurations.

Given this, there are essentially three options:

1. Keep ncrcat, and require it as an external when a user ports the diagnostics to their machine.

2. Replace ncrcat with a python equivalent, which will maintain ease of use, but will likely be slower.

3. Create a namelist option that allows the user to choose between either the ncrcat option or the python replacement, depending on their preference. This keeps the advantages of the previous options, but does increase the complexity of the namelist file and underlying code base.

In averaging_example.py lines 56-59:

        #Average time dimension over time bands, if the bands exist:
        if 'time_bnds' in cam_ts_data:
             cam_ts_data['time'].values = cam_ts_data['time_bnds'].mean(dim='nbnd')
             cam_ts_data = xr.decode_cf(cam_ts_data)

Produce an error with recent xarray version(s) (tested for 0.15.1). The problem is that Xarray doesn't want to modify the values of a coordinate variable. To correct this, I replaced the lines with:

        #Average time dimension over time bounds, if bounds exist:
        if 'time_bnds' in cam_ts_data:
            time = cam_ts_data['time']
            time = xr.DataArray(cam_ts_data['time_bnds'].mean(dim='nbnd').values, dims=time.dims, attrs=time.attrs)
            cam_ts_data['time'] = time 
            cam_ts_data.assign_coords({'time':time})
            cam_ts_data = xr.decode_cf(cam_ts_data)

ADF run type

Model vs. Model

What happened?

There is a syntax error on line ~760 in adf_diag.py that will cause ADF to fail if triggered.

ADF Hash you are using

00c0765

What machine were you running the ADF on?

CISL machine

What python environment were you using?

NPL (CISL machines only)

Extra info

No response

New feature type

Other (please explain below)

What is this new feature?

I tried to use ADF to compute climos from the ne30-L58 runs. I was trying to compute a 10-year climo on the years 1980-1989

The script didn't go anywhere in 12 hours. I talked to Jesse who recommended that I added an issue.

Examples of history files are at:
/glade/scratch/hannay/archive/f.e21.FWscHIST.ne30_L48_BL10_cam6_3_035.tphysac_reorder_zm2_tau7200.001.hf

Assistance required?

Yes, I am requesting a new feature but won't have time to develop it

Extra info

No response

New feature type

Other (please explain below)

What is this new feature?

The sysadmins managing the CGD machines have requested that users install their own version of conda to then use when generating conda environments (e.g. for the ADF). Thus instructions for how to install a local version of conda should be provided, either in README.md or in a separate location that is at least pointed to from the README.md file. This will be done once I have verified that the install workflow works on the CGD machines.

Assistance required?

No, I will make a PR when the feature is ready

Extra info

This issue is mostly a reminder to update the ADF documentation once I have verified that the standard conda install workflow works on the CGD machines. This will have no direct impact on the ADF itself.

Certain external functions, like GeoCAT's interp_hybrid_to_pressure function, can produce a lot of printed messages that are sent to the user's screen, but that generally aren't useful to the user unless they are debugging.

Given this, it would be helpful to develop a new class method in the ADF to prevent all messages from a function unless the debug log is enabled, in which case the messages will be sent there.

One of the most viewed outputs of the AMWG diagnostics is the table of numbers. They just show one variable in each row of the table, with columns of the global average of the variable in the simulation, the reference simulation or observations, the difference, and the RMSE.

I'd like to propose that we could provide similar information more flexibly, and at the same time provide more information.

In the attached prototype code, I am assuming time series files (but not necessarily globally averaged already). It is configured right now as a CLI, and the function main orchestrates the work. It loads the file, gets the variable, checks if there are spatial dimensions (if so, do weighted average), then it does an annual average (assuming that time can be correctly parsed from the time series file), and then calculates:

• mean
• sample size
• standard deviation
• standard error of the mean
• 5%/95% confidence interval of the mean
• linear trend
• p-value of linear trend
  It then puts those into a CSV file (appending if it already exists), and then it converts that CSV file into the simplest HTML file that you can imagine (example with one variable attached).

Note that this has only acted on one simulation! I haven't bothered to deal with the comparison piece. To get the RMSE will take another calculation, but I believe that will be easy to add. Having a CSV for each case might be very useful in the long run, since it would be easy to write scripts that get the stats from these files across many cases.

Adding the standard deviation, standard error, confidence interval, and linear trend analysis should provide a lot of additional value when looking at the diagnostics. Obviously if there are other things, we can add them on.

I have done limited testing with this code, and I didn't have a chance to implement it in the current version of the package. I was hoping that @nusbaume could find a little time to do that if @cecilehannay agrees that this is a viable way to start making the table.

amwg_table_replacement.zip

After the latest DTF meeting, it was determined that it would be better to have all of the test and baseline case variables inside the same config sections, with diag_basic_info containing only very-high level, non-case specific variables.

Along with this change, the scope-defining indentation should be increased from two to four.

Finally, the current config_cam_baseline_example.yaml file should be replace with one that can run "out-of-the-box" on the CISL machines (e.g. Casper), which will help with the upcoming walk-through and hackathon.

ADF run type

Model vs. Model

What happened?

If a user enables time series generation, but includes a variable that is not present in the model history files, then the ADF dies due to an ncrcat error. A better solution that matches the ADF design philosophy would be to have the time series generator simply skip this missing variable, and instead raise a warning notifying the user that the variable is missing (but that the ADF will continue processing the remaining variables).

Implementing this bug fix will likely be easier after Intake-ESM catalog generation has been enabled.

ADF Hash you are using

c621d1d

What machine were you running the ADF on?

CISL machine

What python environment were you using?

ADF-provided Conda env

Extra info

No response

As pointed out by Mariana, the word "namelist" implies a specific Fortran file format, as opposed to the YAML file used in the actual diagnostics. Thus the command-line argument "namelist" and the actual name of the example YAML file should be modified so that it uses a different nomenclature, like "config" or "set-up".

Currently this routine has a fixed ylim of [1000,1] and a linear scale. It would be nice to add optional kwargs to allow a custom pressure range and/or a log pressure scale to be specified. I am not sure how these arguments would get passed all the way down to _zonal_plot_preslat() -- would it be by adding **kwargs to zonal_plot(), plot_zonal_mean_and_save() and zonal_mean()?

It could be beneficial to provide an option to perform compression on the generated time series and (re-gridded) climatology files, along with any other files the diagnostics produces, in case the user needs to save on disk space.

It might also be beneficial to use the tools provided by the ldcpy package to automatically determine which compression method is "best" depending on what the user requires.

The current diagnostics config files make it sound as if the variables start_year and end_year set over which years a climatology will be calculated for a particular model case. However, they in fact only set the length of the generated time series files, which means if someone already has time series files then these variables won't do anything, as the entire time series will be used each time to calculate the climatologies.

To remedy this issue, the create_climo and averaging_example.py script (as well as any future averaging scripts) should be modified to sub-select the time series such that only the user-specified year range is involved in the calculation of the climatologies.

New feature type

New plot and/or plot enhancement

What is this new feature?

Continuing to add variable defaults to adf_variable_defaults.yaml file.
This is waiting on changes from earlier PR #84 to go in first, then I will do a PR for this.

Assistance required?

No, I will make a PR when the feature is ready

Extra info

Next step is to get a list of variables to add....

It would help the usability of the diagnostics package if the config files allowed the user to reference environment (shell) variables in the YAML file itself. For example, instead of:

cam_diag_plot_loc: /some/path/username/plots

One could have:

cam_diag_plot_loc: /some/path/$USER/plots

Where $USER is an environment variable that contains one's username.

Currently the generated zonal mean plots contain neither axis labels nor color bars for the contour maps. These features should thus be added in a future update in order to make the plots more useful to the user.

Currently the conda environment provided for the ADF doesn't specify external package versions, which means the latest/greatest version will be used when the environment is created. This means that if there are upstream conflicts in those packages the ADF can be broken in a way that can be quite difficult to debug by the average user.

Thus to avoid this situation, it would be better to have the default conda environment use specific package versions that are known to work with the ADF. This will allow most users/developers to work with the ADF independent of upstream package changes.

Try calling the CVDP using the CAM diagnostics framework. This will provide an initial test for how easy/hard it is to incorporate external diagnostics packages, and may provide a code base for further development of a unified diagnostics package.

The current YAML config example file (config_cam_baseline_example.yaml) no longer uses the plot_example.py script, plus it is unclear if it really provides any additional insight to a user or developer that global_latlon_map.py doesn't already offer. On top of that, it is another, rarely used script that requires extra maintenance whenever the ADF->script interface is modified (which will be done in issue #66 ).

Given this, it would likely be cleaner and easier to simply remove plot_example.py. If by chance someone needs that particular script then it can always be recovered from older ADF versions.

Currently in the ADF a set of specific variables is passed to each external script. However, this makes certain parts of the internal ADF code quite complex, and makes the python ADF/script interface overly rigid and potentially difficult to work with.

To alleviate this, it might be better to pass either the main ADF object itself, or another script-specific object, into each external script. This would make the ADF/script interface simple and common to essentially all scripts, give the scripts access to every variable that the ADF holds (that isn't private), and would make the ADF lib code itself a lot cleaner. The downside is that a script developer would need to learn how to use this object, but with good documentation that learning effort will hopefully be minimal.

For the discussion that helped initiate this issue, please see: #65

Greetings,

Starting with @brianpm comment on this:

We need a way to specify defaults on a per-variable basis, as @andrewgettelman mentioned, and Adam Phillips brought up from the beginning. We didn’t build any support for this yet, but it could be as simple has having a file (json or yaml or csv or whatever), in which every entry looks something like this:

Variable, scale_factor, to_units, cmap.  , diff_cmap, cnLevels.   , diff_cnLevels, map_projection
PRECT   , 86400000.   , "mm/day", ‘BuGn’ , ‘BrBG’.  , (0, 100, 10), (-20, 20, 2), “Molleweide"

We can decide what fields need to be in there; this was just what occurred to me at this moment. Then plotting scripts could query this table if a flag (“amwg_defaults” ?) is set.

ADF created timeseries should include the date range in the file names. At present, timeseries that are created are named:
$cam_case_name.ncrcat.$variable.nc
To allow users to know the dates within the file, to allow future CVDP integration, and as the ADF will likely be used to form timeseries files (as it's easier than other methods), I propose the files be named the same as standard CESM timeseries files:
$cam_case_name.h0.$variable.YYYYMM-YYYYMM.nc

According to the CAM source code, the SE dycore (and possibly other dycores) have a dry vertical coordinate system, which means that the standard hyxx variables will not produce the full atmospheric pressure, but instead the dry pressure.

Given that the diagnostics package should place all model results on the same pressure levels when doing comparisons or analyses, there should be a way to either interpolate the SE dycore output to the full atmosphere pressure levels, or to convert baseline model variables to dry pressure levels (if need be).

In plot_example.py, line 223 leads to an error if either minval or maxval are equal to 0:

    if np.sign(minval) != np.sign(maxval) and mplv > 2:

because np.sign() will return 0 in those case. That leads to trying to specify either vmin or vmax the same as vcenter for the normalization, which kills matplotlib because they need to be in ascending order.

To get around it, I changed that logic to:

    if ((minval < 0) and (0 < maxval)) and mplv > 2:

It looks like a new python package has been developed at NCAR called "Intake-esm", which is designed to query and load various forms of output from Earth System Models. The documentation for this package can be found here:

https://intake-esm.readthedocs.io/en/latest/

It might be worth seeing if this package can be used in the CAM diagnostics package to improve the reading of CAM (or other model) output files, at least in situations where one can't use NCO operators. Any thoughts?

ADF run type

Model vs. Model

What happened?

A bug happened! not really

ADF Hash you are using

35c75be

What machine were you running the ADF on?

CISL machine

What python environment were you using?

NPL (CISL machines only)

Extra info

I am pretending to have a bug . I will not change code just add a comment.

There are a myriad of different ways a user can configure a plotting script in python. How should the CAM diagnostics manage the desire of a user to modify simple configuration options (e.g. the color bar used) for the diagnostic figures? I can so far think of three options:

1. Do not allow the user to configure the plots through the namelist. If the user wants a plot configured differently, they can simply modify the plotting script directly, or add their own script to the namelist. This is the simplest option from a development and maintenance standpoint, but requires users to know python in order to change anything.

2. Allow all possible configuration options to be controlled via the namelist. This greatly reduces the need of the user to know python in order to configure the scripts to their liking. However, it increases the effort needed to develop and maintain the diagnostics, and may cause the namelist file itself to increase dramatically in size.

3. Allow some configuration options in the namelist, but require all other configuration options to either be determined automatically, or through option 1 (direct modification of the script itself). This would likely be a a trade-off between options 1 and 2, both in terms of advantages and difficulties.

Any thoughts or opinions?

Currently the diagnostics package can only process one simulation (case) at a time. However, as brought up by @andrewgettelman, there are various scientific reasons why it would be helpful to process multiple cases all at once, compare them all to the same observations or baseline, and then plot them all on the same figure.

In order to facilitate this new capability, the diagnostics package should be able to take in a list of cases that can then all be processed during the same diagnostics run, thus providing a single data collection that can be used to generated the relevant multi-case figures.

ADF run type

Model vs. Model

What happened?

For the plot_map_and_save function in plotting_functions.py, the 'mpl' (MatPlotLib) passing attributes functionality was not properly initialized. This is corrected in this PR, and an example of adding labels for units to the colorbar in the map plots is illustrated in adf_variable_defaults.

ADF Hash you are using

35c75be

What machine were you running the ADF on?

CISL machine

What python environment were you using?

NPL (CISL machines only)

Extra info

Borderline between bug fix and a feature (only hit if feature is enabled)