Graph is empty after update to 2.9.0 on some DAGs about airflow HOT 5 CLOSED

@bbovenzi Yes, that seems to be the case. My guess is this didn't happen before because Airflow was not drawing datasets in the graph

from airflow.

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

from airflow.

If it helps to decipher this problem, i found another case:

And in this one, i was able to find old tasks with that name (That task does not exist any more) - However even after deleting the deprecated task instances using the UI, the same error persists in the console, my impression is it seems some "phantom records" are being picked up by the graph library?.

Upon further investigation, it seems this might be caused by the new datasets viz in the graph? Because i see that we get this record that is not present in the datasets UI, because it's from a deprecated table:

    {
      "consuming_dags": [],
      "created_at": "2023-10-16T18:24:02.812802+00:00",
      "extra": {},
      "id": 1081,
      "producing_tasks": [
        {
          "created_at": "2023-10-16T18:24:02.827186+00:00",
          "dag_id": "dbt_pipeline_agg_alb_logs",
          "task_id": "dbt_models.raw_alb_logs_api_requests_categorized",
          "updated_at": "2023-10-16T18:24:02.827200+00:00"
        }
      ],
      "updated_at": "2023-10-16T23:11:20.408211+00:00",
      "uri": "raw_alb_logs_api_requests_categorized"
    }

(This is from /api/v1/datasets?dag_ids=dbt_pipeline_agg_alb_logs)

Is there a way to clean it up? Or to make Airflow ignore this record instead of breaking the graph?

from airflow.

cc @bbovenzi

from airflow.

So this only happens to DAGs with Datasets? And it sounds like it happens when we retrieve datasets but either the dataset or task no longer exists so we're pointing to a nonexistent node. Sounds straightforward enough to add a check for that.

from airflow.