mbkraus / incremental_training Goto Github PK

I always get this issue and the DAGs are never updated. Any idea to resolve the issue ?

Getting below error

` File "/usr/local/airflow/.local/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in new
airflow_container | _message.Message._CheckCalledFromGeneratedFile()
airflow_container | TypeError: Descriptors cannot not be created directly.
airflow_container | If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
airflow_container | If you cannot immediately regenerate your protos, some other possible workarounds are:
airflow_container | 1. Downgrade the protobuf package to 3.20.x or lower.
airflow_container | 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

2022-10-18 14:10:17.307792: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
airflow_container | 2022-10-18 14:10:17.308008: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
airflow_container | [2022-10-18 14:10:17 +0000] [172] [INFO] Handling signal: ttou
airflow_container | [2022-10-18 14:10:17 +0000] [3658] [INFO] Worker exiting (pid: 3658)
airflow_container | 2022-10-18 14:10:18.282079: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
airflow_container | 2022-10-18 14:10:18.282110: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
airflow_container | 2022-10-18 14:10:19.312501: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory`

My containers are up and running fine.

srpatil$ docker-compose -f docker-compose-project.yml up
Creating network "incremental_training_default" with the default driver
Creating postgres_container  ... done
Creating zookeeper_container ... done
Creating mlflow_container    ... done
Creating kafka_container     ... done
Creating airflow_container   ... done

Looking at the terminal, I can see that my DAGs are getting triggered correctly and the model is getting updated, but I just cannot see any parameters getting logged in MLFlow.

airflow_container | Using TensorFlow backend.
airflow_container | [2020-02-28 08:19:10 +0000] [186] [INFO] Handling signal: ttou
airflow_container | [2020-02-28 08:19:10 +0000] [8320] [INFO] Worker exiting (pid: 8320)
airflow_container | [2020-02-28 08:19:41 +0000] [186] [INFO] Handling signal: ttin
airflow_container | [2020-02-28 08:19:41 +0000] [9339] [INFO] Booting worker with pid: 9339
airflow_container | [2020-02-28 08:19:41,584] {{__init__.py:51}} INFO - Using executor LocalExecutor
airflow_container | [2020-02-28 08:19:41,585] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
airflow_container | Using TensorFlow backend.
airflow_container | [2020-02-28 08:19:45 +0000] [186] [INFO] Handling signal: ttou
airflow_container | [2020-02-28 08:19:45 +0000] [8525] [INFO] Worker exiting (pid: 8525)
airflow_container | [2020-02-28 08:20:15 +0000] [186] [INFO] Handling signal: ttin
airflow_container | [2020-02-28 08:20:15 +0000] [9552] [INFO] Booting worker with pid: 9552
airflow_container | [2020-02-28 08:20:16,656] {{__init__.py:51}} INFO - Using executor LocalExecutor
airflow_container | [2020-02-28 08:20:16,657] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/ai

Hello,

I would like to ask how to adapt the code in order to allow the model to train using GPU ?

Thank you in advance.