feluelle / airflow-diagrams Goto Github PK

As the title says, I added airflow-diagrams as our user in the "who uses it?" section in the diagrams README.

Thank you for building it :)

I get an ScannerError saying ScannerError: mapping values are not allowed here in "file.py", line 18, column 12 when i try to airflow-diagrams generate --from-file file.py'. I wonder why this happends, because every DAG file has at least default_args = {
"owner": "myself",
"depends_on_past": False, ...` in it, which looks like the source of the problem to me. Every if statement in the DAG file throws an error, because it uses a colon. Is this also a problem by not using --from-file ?

If we allow reading Airflow information from a file, the command generate does not need direct access to the Airflow REST-API, which makes it more secure as the REST API does not need to be enabled only to use this feature.

Add a separate command for only retrieving airflow information and let it store the information to a file.

Airflow DAGs that utilise TaskGroups would be better represented if we were to make use of Clusters

Airflow "Transfer" operators could be created differently:

E.g.

• two nodes ie. from & to
• no node and add the label to an edge (between nodes)

Diagrams Cluster can nicely group nodes together. This for example can be useful for something like Airflow TaskGroups.

We would like to have some examples for generated (and rendered) diagrams. It would be nice to have some complex dags, but also small ones would work for displaying in the README.

We can upload them to the examples directory and link them in the README.md.

I have a DAG as .py which i try to render using airflow-diagrams generate --from-file path/file.py. The file has comments in it.
The error i get is:
ParserError: expected '<document start>', but found '<scalar>' in "file.py", line 13, column 1

I believe it is necessary to adjust the read file. Delete lines which begin with #, also having in my to .lstrip the line because of indentation.

If you have a DAG which contains tasks where the id contains either numbers or dashes then, although the diagram Python file is created, it cannot be run to generate the diagram image.

e.g.

from diagrams import Diagram
from diagrams.generic.blank import Blank

with Diagram("airbyte", show=False):
    airbyte-job_sensor = Blank("airbyte_job_sensor")
    airbyte_trigger_async = Blank("airbyte_trigger_async")
    
    airbyte_trigger_async >> [airbyte_job_sensor]

➜  airflow-diagrams git:(master) ✗ python3 examples/airbyte_diagrams.py
  File "/Users/nathan/Projects/airflow-diagrams/examples/airbyte_diagrams.py", line 5
    airbyte-job_sensor = Blank("airbyte_job_sensor")
    ^
SyntaxError: cannot assign to operator

I installed airflow-diagrams as per the instructions (FYI - I also had to install Graphviz, currently running 2.50.0) but just trying to run the example diagrams the returned image seemed to be lacking in detail.