Comments (13)
@maddenp-noaa Thanks for getting to this quickly! I believe that error is expected given the minimal example I provided. Unfortunately I have the "Hera is down" blues today and that's where my work is staged so I will have to wait until tomorrow to check that it works in the context of the SRW branch I'm working on.
from uwtools.
Hi @mkavulich, and thank you for the report.
I see what's happening and am discussing with the uwtools
team. I suspect that this is not a bug per se, in that it's not something that I'd expect to work "out of the box" the way something like {{ 2 + 2 }}
should. It requires specific configuration in the Python application code to inform Jinja about where to find template files (and even that macros.jinja
should be interpreted as a file), and users would need a way to specify one or more paths to search for files, which uw
doesn't currently provide. But it sure looks like a valuable feature request!
I haven't used Jinja's import
in the past, so the foregoing is based on reading and tests I've just done. If any of it sounds wrong to you, please let me know.
More soon...
from uwtools.
We have a plan to implement this feature. (For readers with EPIC Jira access: UW-523.) We'll try to get it in our upcoming v2.1.0 release.
If you're using uwtools
in the context of a UFS App (e.g. SRW), it may take longer for the new version to be explicitly added there, but we can help individuals with one-off updates, etc.
If you're using uwtools
as a standalone tool, you'd be able to build your own version from a git clone as soon as we merge a fix to main
, which we can also help with. We'll close this issue when that happens, so you'll know.
from uwtools.
@maddenp-noaa I just wanted to provide some background info. This issue arose when I tried to use jinja2 macros with a relatively recent version of the uwtools CLI in the SRW App. Here's the PR that gives a lot more detail. The odd thing is that an older version of uwtools (not a CLI; it was just the uwtools repo cloned within the SRW App) was working with the macros. When I updated my PR to the latest develop branch of the SRW that brought with it the conda installation of various python packages and the uwtools CLI, the macros no longer worked and I got the behavior @mkavulich described above. Part (maybe all) of the problem might be a mismatch between the version of jinja2 that uwtools expects (the one in the conda installation) and the one that the SRW App uses in spack-stack-1.5.0. If interested, you can read the details of what I've tried in that PR. I'm happy to provide more info if that will help. Thanks.
from uwtools.
@gsketefian Thanks for this, very helpful. I'll look at the older uwtools
code to see what it was doing to support loadable templates.
from uwtools.
Hi @maddenp-noaa FYI the SRW App code manager Michael Leuken ran a test of my PR #1005 using the version of jinja2 that the uwtools CLI expects, and the failure is still there. The PR has more details of what he did. So apparently it's not (just?) a version issue.
from uwtools.
I believe I see what needs to be done in uwtools
to support loading Jinja2 macros. It should be straightforward to implement, and I can probably do so on Monday. I'll see if there's an easy way to update the reproducer environment @mkavulich detailed above so that you can make use of the fix without waiting for an updated uwtools
version to be incorporated into SRW.
from uwtools.
I have a potential update to uwtools
to address this issue, and have installed it into the SRW I built on hera
in /scratch2/BMC/wrfruc/Paul.Madden/gh422/ufs-srweather-app
. After doing the prescribed module load
and conda activate
commands, and omitting debug and stacktrace info, I get
% uw template render -i GridStat_or_PointStat.conf -o out -v --values-file met_plus_settings.yaml
...
The list of valid field groups for forecasts (fgs_fcst) must be identical
to that for observations (fgs_obs) but isn't:
fgs_fcst = ['ASNOW', 'REFC', 'RETOP']
fgs_obs = ['APCP', 'ASNOW', 'REFC', 'RETOP']
Is this the expected behavior given these files? It seems intentional. Please let me know if not, but otherwise please feel free to test with my SRW install. I've tried to open up permissions sufficiently. If things look ok, I could give you a recipe for updating your own SRW install with this uwtools
version. This version is based on the latest release, though, so it may be that other uwtools
uses in SRW may need to be updated to continue to work correctly.
The update works as follows: By default, the directory containing the template file (identified by -i
in the above command) will be searched for template files named in e.g. Jinja2 import
statements. Alternatively, an optional --search-path
CLI option can be used to suppress the default path and to provide a colon-separated series of paths to search for templates.
from uwtools.
@maddenp-noaa Yes, thank you for coming up with a fix so soon! I'm assuming you didn't run one of the workflow-end-to-end (WE2E) tests in the SRW App to get that message; you just ran that command on the command line. If so, that would explain the error message, which (if I remember right) is coming from the jinja2 macros file.
I would like to test your fix on my PR (#1005) when Hera is back up. Will it work if I simply replace the uwtools directory in the conda build that the SRW requires with your version? Or are there other packages to update as well?
If the update requires changes in other calls to uwtools in the SRW, it may be that we'll have to first open a new PR into the app to update uwtools (along with appropriate tests), and, once that's merged, re-test my PR. Still, I'd like to test my PR with your fix to see whether it at least works for the WE2E tests that I'm running, so it would be great if you could send that recipe for updating uwtools. Thanks!
from uwtools.
@gsketefian You should be able to test by replacing any conda activate srw_app
command with conda activate /scratch2/BMC/wrfruc/Paul.Madden/gh422/ufs-srweather-app/conda/envs/srw_app/
to activate my environment directly. If that's done in one place (or a few easily updatable places) in your automation, that might be sufficient for testing.
Alternatively, I think you could rename your ufs-srweather-app/conda
(following the naming conventions in the initial issue report) directory and then ln -s /scratch2/BMC/wrfruc/Paul.Madden/gh422/ufs-srweather-app/conda ufs-srweather-app/conda
to use my entire conda installation instead. (conda is sensitive to paths due to the way it encodes absolute paths into binaries, and you can't e.g. copy one conda installation to a new path and expect it to work, but I think a symllink should work.)
I think one of those would be easiest to try first, but I can give you an recipe for updating your own installation with the test uwtools
version if we're initially unsuccessful. Please let me know how it goes.
from uwtools.
We've incorporated this fix into today's v2.1.0 release. You can install this version into an existing, activated conda environment with
conda install -c ufs-community -c conda-forge uwtools=2.1.0
As before, it's possible that other CLI invocations of or API calls into uwtools
from SRW will need to be (hopefully lightly) updated for compatibility with the new release, and someone on our team will be working on an SRW PR to that effect.
from uwtools.
From following ticket traffic elsewhere, it sounds like this is working as expected. If I don't hear objections in the next couple days, I'll close this ticket. Thanks again for the report.
from uwtools.
@maddenp-noaa FYI Gerard was on vacation this week so I assume he hasn't had a chance to test it for his PR yet.
from uwtools.
Related Issues (20)
- Rocoto JSONSchema Metatasks var requirement bug (PR UW-381) HOT 1
- Standardize YAML keys on hyphens over underscores. HOT 1
- YAML SafeLoader converts multi-day time deltas to integers HOT 2
- UW-334 add a modal CLI support for the Rocoto XML tool HOT 1
- UW-322 validate a rocoto xml workflow HOT 1
- Forecast run method needs refactor.
- UW-413 Ensure CLI functions accept file and stdin streams for both input and output HOT 1
- UW-412: expand tests to also not provide optional args HOT 1
- UW-370: consider merging j2template and templater modules HOT 1
- Handle `<dependency>` element possibilities. HOT 1
- UW-418: Split .ini & .sh handling HOT 1
- UW-420: Forbid level-incompatible config transformations HOT 1
- UW-479: add uw --debug CLI option HOT 1
- Error trying to reach documentation HOT 6
- translate does not preserve indentation HOT 7
- UW-500 - Allow forced conversion between formats HOT 5
- UW-504 - Allow YAML as supplemental config always HOT 1
- uwtools.api.file.copy cannot create nested directories HOT 1
- datetime.UTC does not work in Python < 3.11 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uwtools.