primary_key incorrect - causes failures in target-postgres, and target-snowflake (probably more) about tap-googleads HOT 7 CLOSED

Added the error messages to the issue

from tap-googleads.

I'm getting a very similar error with target-snowflake transferwise variant.

2022-03-18T16:36:51.571689Z [info     ] invalid identifier '"CUSTOMER_CLIENT.ID"' cmd_type=loader job_id=gads_1 name=target-snowflake run_id=dc70b619-bb7d-4a3e-97c7-8bc53a27fca2 stdio=stderr

which pretty much immediately follows this log:

2022-03-18T16:36:51.123275Z [info     ] time=2022-03-18 10:36:51 name=target_snowflake level=INFO message=Running query: 'CREATE TABLE IF NOT EXISTS gads."CUSTOMER_HIERARCHYSTREAM" ("_SDC_BATCHED_AT" timestamp_ntz, "_SDC_DELETED_AT" text, "_SDC_EXTRACTED_AT" timestamp_ntz, "_SDC_SOURCE_SCHEMA" text, "CUSTOMERCLIENT" variant, PRIMARY KEY ("CUSTOMER_CLIENT.ID")) data_retention_time_in_days = 1 ' with Params {'LAST_QID': None} cmd_type=loader job_id=gads_1 name=target-snowflake run_id=dc70b619-bb7d-4a3e-97c7-8bc53a27fca2 stdio=stderr

This, to me, indicates that the issue is not that the primary key is wrong but that the primary key is located inside of a variant or json field and therefore the primary key can't be accessed for the purposes of table creation.

from tap-googleads.

Could you post the error you're getting from target-postgres?

from tap-googleads.

One of the errors is due to: https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/streams.py#L81

• This one we just need to add the correct primary key if this isn't it (not sure how it was missed)

The other is due to: https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/streams.py#L32

• Need to add the correct primary key here

• Final note is we should add a Test to run the whole connector with Target-Postgres as that's a pretty good standard. Today where this is running in production uses target-s3 so a lot of requirements are looser

from tap-googleads.

Let me know if this doesn't' work now and I'll reopen this!

from tap-googleads.

Thanks a bunch to @jlloyd-widen

from tap-googleads.