opendatahub-io-contrib / docker-airflow Goto Github PK
View Code? Open in Web Editor NEWThis project forked from barney-s/docker-airflow
Docker Apache Airflow
License: Apache License 2.0
This project forked from barney-s/docker-airflow
Docker Apache Airflow
License: Apache License 2.0
Hey, Kebechet!
Create a new patch release, please.
When using the Kubernetes executor, the airflowui and scheduler fail to start:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1013, in scheduler
do_pickle=args.do_pickle)
File "<string>", line 4, in __init__
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 380, in __init__
super(SchedulerJob, self).__init__(*args, **kwargs)
File "<string>", line 6, in __init__
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 86, in __init__
self.executor = executor or executors.get_default_executor()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
DEFAULT_EXECUTOR = _get_executor(executor_name)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 87, in _get_executor
return KubernetesExecutor()
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 715, in __init__
self.kube_config = KubeConfig()
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 284, in __init__
self.kube_client_request_args = json.loads(kube_client_request_args)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Trying to create an Airflow cluster using the following resource manifest:
apiVersion: airflow.apache.org/v1alpha1
kind: AirflowCluster
metadata:
name: pk-cluster
spec:
executor: Kubernetes
ui:
replicas: 1
image: "quay.io/opendatahub/docker-airflow"
version: "openshift"
scheduler:
image: "quay.io/opendatahub/docker-airflow"
version: "openshift"
worker:
image: "quay.io/opendatahub/docker-airflow"
version: "openshift"
dags:
subdir: "airflow/example_dags/"
git:
repo: "https://github.com/apache/airflow/"
once: true
branch: master
airflowbase:
name: pc-base
The airflowui and scheduler components fail to start:
NAME READY STATUS RESTARTS AGE
pod/airflow-on-k8s-operator-controller-manager-69f6459f-j24tq 1/1 Running 0 3h13m
pod/pc-base-postgres-0 1/1 Running 0 82m
pod/pk-cluster-airflowui-0 0/1 Error 9 25m
pod/pk-cluster-scheduler-0 1/2 CrashLoopBackOff 9 25m
They fail because the Python kubernetes library is not included in the container image:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1013, in scheduler
do_pickle=args.do_pickle)
File "<string>", line 4, in __init__
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 380, in __init__
super(SchedulerJob, self).__init__(*args, **kwargs)
File "<string>", line 6, in __init__
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 86, in __init__
self.executor = executor or executors.get_default_executor()
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
DEFAULT_EXECUTOR = _get_executor(executor_name)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 86, in _get_executor
from airflow.contrib.executors.kubernetes_executor import KubernetesExecutor
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 30, in <module>
import kubernetes
ModuleNotFoundError: No module named 'kubernetes'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.