Redshift Ingestion broken in 0.13.2 about datahub HOT 4 OPEN

@joshua-pgatour what CLI version is this with?

I merged a fix related to this in #9967

from datahub.

Thank you for the reply. I have tried going back one version at a time on datahub-actions dockerhub releases and the KeyError seems to stop happening around v10. However, I still have a memory issue. I have pretty much maxed out the size my pod can be and it still fails with memory SIGKILL. Any suggestions on getting around this? Here is my current recipe:

`source:
type: redshift
config:
host_port: ''
database: pgat
username:
table_lineage_mode: mixed
include_table_lineage: false
include_tables: true
include_views: false
profiling:
enabled: false
profile_table_level_only: false
stateful_ingestion:
enabled: true
password: '${redshift_secret2}'
schema_pattern:
allow:
- 'curated_.*'

`

from datahub.

So I figured out how to change the CLI version in the ingest recipe. I'm sorry I thought it was controlled by the datahub-actions container version. (Didn't know it was controlled in the recipe). 0.10.5.1 CLI works fine on 16gb memory and there is no KeyError. I will experiment at what point this breaks. But I gotta believe there's a memory leak in newer versions.

from datahub.

I can confirm that v0.13.2 has the memory problem. v0.13.1 works, but in my testing the ingest process has slowed significantly since 0.12

from datahub.