godatadriven / whirl Goto Github PK

would it make sense to move some of this to global variables in this script? Would reduce the amount of change required if we ever need/want to change username/password/api endpoint?

Originally posted by @danielvdende in #74 (comment)

Right now, we install complete Spark application in the Airflow container.
We want to have Spark run in a separate container.
We only need the client/driver for Spark in the Airflow container

Would it make sense to also add an example where we actively add config to dags with a policy? Perhaps something like:

def task_assign_default_owner(task: BaseOperator):
    if not task.owner or task.owner.lower():
        task.owner = "baboon"

Originally posted by @danielvdende in #74 (comment)

https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html#deferrable-operators-triggers

2 examples:

• using a provided operator/sensor
• writing a custom deferrable operator

Feature request:
Users should be able to specify where whirl looks for environment directories. eg -d to specify the folder that contains the environments (defaults to SCRIPT_DIR)

Use cases:
Embedding the environments in source control with DAGs to allow test environments to move around with code.

Describe what Whirl is and how to use it

Hi @bjgbeelen
Can you advise below error. This is a first time examining with whirl for the first example "SSH to Localhost".
There are couple of errors "No module named 'werkzeug.wrappers.json';" at step 29/29 and then exit with code 1.

airflow_1  | ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package
local-ssh_airflow_1 exited with code 1

hi @krisgeus
sorry for that I am writing you here actually that Divolte repository has been archived by the owner, and I am stuck in one problem of Divolte session Id

1. every page refresh I am getting new session id, why session id wont be persist till their session timeout configured ?
2. How can we make a function so that we can close this session id manually without waiting for inactivity ?

I am facing these issues
can you please guide me how can we solve this
This would be really helpful for me

Thanks

Airflow removed the AIRFLOW_GID parameter causing the whirl build to break for airflow version 2.2+ - relevant airflow commit here and failing whirl line here

Exact error message when setting airflow_version=2.2.2:

 => [ 3/10] RUN mkdir -p "/etc/airflow/whirl.setup.d/dag.d"                                                                                                                                                                                                      0.3s
 => ERROR [ 4/10] RUN chown -R airflow:airflow "/etc/airflow/whirl.setup.d"                                                                                                                                                                                      0.3s
------
 > [ 4/10] RUN chown -R airflow:airflow "/etc/airflow/whirl.setup.d":
#7 0.246 chown: invalid group: 'airflow:airflow'
------
executor failed running [/bin/bash -o pipefail -c chown -R airflow:airflow "${WHIRL_SETUP_FOLDER}"]: exit code: 1

This is fixed by updating the whirl python docker file as so

RUN chown -R airflow:0 "${WHIRL_SETUP_FOLDER}"

Happy to help remediate!

Let's see if we can use Whirl in our CI to actually run a DAG and test that the DagRun turned successful

provide environment, start/stop commands.

e.g.

whirl -e default (start)
whirl -e default start (start daemon)
whirl -e default stop (stop)

When trying to run an example I am getting an error:

cd $WHIRL_DIR/examples/localhost-ssh-example
whirl start -e local-ssh

Package openjdk-8-jre-headless is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openjdk-8-jre-headless' has no installation candidate
The command '/bin/sh -c mkdir -p /usr/share/man/man1   && update-ca-certificates -f   && apt-get update   && apt-get install -y --no-install-recommends --reinstall build-essential   && apt-get install -y --no-install-recommends --allow-unauthenticated      software-properties-common      wget      dnsutils      vim      git      default-libmysqlclient-dev      gcc      openjdk-8-jre-headless      ${AIRFLOW_BUILD_DEPS}      ${AIRFLOW_APT_DEPS}      nginx      gosu      sudo   && apt-get clean   && apt-get autoremove -yqq --purge   && rm -rf /var/cache/apk/* /var/lib/apt/lists/*   && (find /usr/share/doc -type f -and -not -name copyright -print0 | xargs -0 -r rm)' returned a non-zero code: 100
Pulling airflow (docker-whirl-airflow:py-3.6-local)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling airflow (docker-whirl-airflow:py-3.6-local)...
ERROR: pull access denied for docker-whirl-airflow, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Right now this is hardcoded to the filename dag.py. See if we can check the contents of the file to detect a DAG