Code Monkey home page Code Monkey logo

docker-airflow's People

Contributors

ajhodges avatar arihantsurana avatar ashb avatar barney-s avatar darkk avatar diraol avatar edrzmr avatar eshizhan avatar hadsed-genapsys avatar ianburrell avatar javioverflow avatar jbdalido avatar jie8357ioii avatar jmcarp avatar kaxil avatar kristi avatar lamroger avatar maxcountryman avatar mendhak avatar mhousley avatar msn1444 avatar noseka1 avatar odannyc avatar puckel avatar rootcss avatar ryanrussell avatar theodoresiu avatar ttaschke avatar vedantmahabaleshwarkar avatar vpavlin avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

docker-airflow's Issues

Expecting property name enclosed in double quotes

When using the Kubernetes executor, the airflowui and scheduler fail to start:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1013, in scheduler
    do_pickle=args.do_pickle)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 380, in __init__
    super(SchedulerJob, self).__init__(*args, **kwargs)
  File "<string>", line 6, in __init__
  File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 86, in __init__
    self.executor = executor or executors.get_default_executor()
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
    DEFAULT_EXECUTOR = _get_executor(executor_name)
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 87, in _get_executor
    return KubernetesExecutor()
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 715, in __init__
    self.kube_config = KubeConfig()
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 284, in __init__
    self.kube_client_request_args = json.loads(kube_client_request_args)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

No module named 'kubernetes'

Trying to create an Airflow cluster using the following resource manifest:

apiVersion: airflow.apache.org/v1alpha1
kind: AirflowCluster
metadata:
  name: pk-cluster
spec:
  executor: Kubernetes
  ui:
    replicas: 1
    image: "quay.io/opendatahub/docker-airflow"
    version: "openshift"
  scheduler:
    image: "quay.io/opendatahub/docker-airflow"
    version: "openshift"
  worker:
    image: "quay.io/opendatahub/docker-airflow"
    version: "openshift"
  dags:
    subdir: "airflow/example_dags/"
    git:
      repo: "https://github.com/apache/airflow/"
      once: true
      branch: master
  airflowbase:
    name: pc-base

The airflowui and scheduler components fail to start:

NAME                                                            READY   STATUS             RESTARTS   AGE
pod/airflow-on-k8s-operator-controller-manager-69f6459f-j24tq   1/1     Running            0          3h13m
pod/pc-base-postgres-0                                          1/1     Running            0          82m
pod/pk-cluster-airflowui-0                                      0/1     Error              9          25m
pod/pk-cluster-scheduler-0                                      1/2     CrashLoopBackOff   9          25m

They fail because the Python kubernetes library is not included in the container image:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1013, in scheduler
    do_pickle=args.do_pickle)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 380, in __init__
    super(SchedulerJob, self).__init__(*args, **kwargs)
  File "<string>", line 6, in __init__
  File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 86, in __init__
    self.executor = executor or executors.get_default_executor()
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
    DEFAULT_EXECUTOR = _get_executor(executor_name)
  File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 86, in _get_executor
    from airflow.contrib.executors.kubernetes_executor import KubernetesExecutor
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 30, in <module>
    import kubernetes
ModuleNotFoundError: No module named 'kubernetes'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.