Comments (4)
@bjgbeelen this feature would be great! really looking forward to this
from whirl.
@ilyanoskov @bjgbeelen #61 adds airflow 2.x style unpause and dagrun status checks.
The command ../../whirl -e postgres-s3-spark ci
from within the examples/spark-s3-to-postgres/
directory:
- builds the images
- starts the environment
- waits until the airlfow UI is available
- unpauses the dag (dag_id is retrieved from the dag.py file)
- checks the running state of the dag in a loop
- checks the final dagrun state (
success
orfailure
) and exits with status0
(success) or1
(failure) - stops the environment
from whirl.
It is tested locally on my mac and works at least for this dag and environment. I'm planning to check all other examples too.
Next step is #34 to add github actions to the repo to prove this works ;-)
from whirl.
Closed with #64
from whirl.
Related Issues (20)
- Add GoogleCloudStorage example
- Add custom Python module example
- Add commandline arguments for Whirl HOT 1
- Add License
- Add .gitattributes
- Make whirl .env aware HOT 2
- Add CI to the repo HOT 1
- Make Airflow version configurable HOT 1
- No module named 'werkzeug.wrappers.json' error HOT 2
- Whirl fails to build base image HOT 4
- Allow users to specify the folder to look for environments HOT 1
- Add >2.1.x Support HOT 2
- Add examples around deferrable operators and triggers
- Add an example where we actively add config to dags with a policy
- Move some hardcoded api endpoint values to global variables in whirl script
- Create quick start and more documentation to describe the goal and use of whirl
- Fix deprecation warnings
- Make it easier to install and use whirl
- Divolte issue HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from whirl.