tap-googleads's People
Contributors
Stargazers
Watchers
Forkers
danielpdwalker hotgluexyz widen marlettefunding tomekzbrozek gary-beautypie dadosfera sebastianswms patrickrobothamtap-googleads's Issues
Customer Not Enabled - If more than 25% of customers fail, the whole job should fail
https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/client.py#L93
Right now we just warn if we get this issue, but if more than 25% of customers have this failure we should really fail the whole tap.
Service Account
- Had some issues with accounts expiring
- Would like to add service accounts feature to this
start_date should be less than end_date
If this isn't true you won't get results you would expect (Not all data). We should force this by throwing an exception if that's not the case!
Pagination
nextPageToken implement
400 Client Error: Bad Request for path: /customers:listAccessibleCustomers
Hi,
First of all, thanks for the tap! I've been trying to set it up and bumped into an issue fairly early on. I'm seeing singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /customers:listAccessibleCustomers.
In the API console, I can see the google.ads.googleads.v8.services.CustomerService.ListAccessibleCustomers requests erroring out at a 100% rate.
Mind you, I haven't had an AdWords tap setup previously, this is my first attempt and right now it seems you can't even enable the AdWords endpoints through the console anymore.
Below is a relevant section of the run, happy to provide more details.
2022-04-06T17:31:33.245485Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=tap-googleads v0.0.8, Meltano SDK v0.3.17) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.250424Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Skipping parse of env var settings... cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.252183Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Config validation passed with 0 errors and 0 warnings. cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.259158Z [info ] time=2022-04-06 17:31:33 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.298158Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.299411Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'adgroups' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.300673Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'adgroupsperformance' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.301823Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'customer_hierarchystream' as child stream to 'accessible_customers' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.304356Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign_performance' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.305435Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign_performance_by_age_range_and_device' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.306853Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign_performance_by_gender_and_device' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.309275Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign_performance_by_location' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.312789Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'conversion_by_location' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.314891Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Added 'campaign_label' as child stream to 'customer_hierarchystream' cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.316696Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Beginning full_table sync of 'accessible_customers'... cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.319023Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.320395Z [debug ] {"type": "SCHEMA", "stream": "accessible_customers", "schema": {"properties": {"resourceNames": {"items": {"type": ["string"]}, "type": ["array", "null"]}}, "type": "object"}, "key_properties": []} cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads (out) run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stdout
2022-04-06T17:31:33.490997Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=OAuth authorization attempt was successful. cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.733141Z [info ] time=2022-04-06 17:31:33 name=tap-googleads level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.239989, 'tags': {'endpoint': '/customers:listAccessibleCustomers', 'http_status_code': 400, 'status': 'failed', 'url': '/v8/customers:listAccessibleCustomers'}} cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.772301Z [info ] Traceback (most recent call last): cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.774294Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/bin/tap-googleads", line 8, in <module> cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.775389Z [info ] sys.exit(TapGoogleAds.cli()) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.777227Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.779541Z [info ] return self.main(*args, **kwargs) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.782108Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.784585Z [info ] rv = self.invoke(ctx) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.786501Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.788280Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.789610Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.791425Z [info ] return __callback(*args, **kwargs) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.792669Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 476, in cli cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.794325Z [info ] tap.sync_all() cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.796105Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 345, in sync_all cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.797660Z [info ] stream.sync() cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.799520Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1010, in sync cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.802393Z [info ] self._sync_records(context) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.803761Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 937, in _sync_records cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.805264Z [info ] for record_result in self.get_records(current_context): cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.806816Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/tap_googleads/client.py", line 123, in get_records cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.808242Z [info ] for record in self.request_records(context): cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.809775Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 293, in request_records cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.811444Z [info ] resp = decorated_request(prepared_request, context) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.813973Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.815616Z [info ] ret = target(*args, **kwargs) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.817097Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/singer_sdk/streams/rest.py", line 209, in _request cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.819078Z [info ] self.validate_response(response) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.820978Z [info ] File "/project/.meltano/extractors/tap-googleads/venv/lib/python3.8/site-packages/tap_googleads/client.py", line 102, in validate_response cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.822722Z [info ] raise FatalAPIError(msg) cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.824592Z [info ] singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /customers:listAccessibleCustomers cmd_type=extractor job_id=2022-04-06T173122--tap-googleads--target-redshift name=tap-googleads run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:31:33.919382Z [info ] time=2022-04-06 17:31:33 name=target_redshift level=INFO message=Getting catalog objects from table cache... cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.034463Z [info ] Traceback (most recent call last): cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.039760Z [info ] File "/project/.meltano/loaders/target-redshift/venv/bin/target-redshift", line 8, in <module> cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.041302Z [info ] sys.exit(main()) cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.042714Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/target_redshift/__init__.py", line 444, in main cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.044336Z [info ] table_cache = load_table_cache(config) cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.045726Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/target_redshift/__init__.py", line 108, in load_table_cache cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.047039Z [info ] table_cache = db.get_table_columns( cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.048219Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/target_redshift/db_sync.py", line 632, in get_table_columns cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.049778Z [info ] return self.query(sql) cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.051377Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/target_redshift/db_sync.py", line 333, in query cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.052934Z [info ] with self.open_connection() as connection: cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.054903Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/target_redshift/db_sync.py", line 329, in open_connection cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.056490Z [info ] return psycopg2.connect(conn_string) cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.057939Z [info ] File "/project/.meltano/loaders/target-redshift/venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.059401Z [info ] conn = _connect(dsn, connection_factory=connection_factory, **kwasync) cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.062110Z [info ] psycopg2.OperationalError: could not connect to server: Connection refused cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.063313Z [info ] Is the server running on host "reportiv.chy9qh3mxgx9.us-east-1.redshift.amazonaws.com" (52.0.140.187) and accepting cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.064663Z [info ] TCP/IP connections on port 5439? cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.066115Z [info ] cmd_type=loader job_id=2022-04-06T173122--tap-googleads--target-redshift name=target-redshift run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7 stdio=stderr
2022-04-06T17:32:50.211361Z [debug ] Deleted configuration at /project/.meltano/run/elt/2022-04-06T173122--tap-googleads--target-redshift/aa915d62-7ead-4578-9e29-6481ff6c22f7/target.d87709ee-3f0d-4e09-a13b-056579b41edf.config.json
2022-04-06T17:32:50.215714Z [debug ] Deleted configuration at /project/.meltano/run/elt/2022-04-06T173122--tap-googleads--target-redshift/aa915d62-7ead-4578-9e29-6481ff6c22f7/tap.42ea2ef2-7198-4888-8e81-172376955ec1.config.json
2022-04-06T17:32:50.216957Z [error ] Extraction failed code=1 job_id=2022-04-06T173122--tap-googleads--target-redshift message=singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /customers:listAccessibleCustomers name=meltano run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7
2022-04-06T17:32:50.218226Z [error ] Loading failed code=1 job_id=2022-04-06T173122--tap-googleads--target-redshift message=(see above) name=meltano run_id=aa915d62-7ead-4578-9e29-6481ff6c22f7
2022-04-06T17:32:50.223251Z [debug ] ELT could not be completed: Extractor and loader failed.
API Version update to 14
Update to version 14
At least these things are broken right now:
- Unrecognized field in the query
response.json() {'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.ads.googleads.v13.errors.GoogleAdsFailure', 'errors': [{'errorCode': {'queryError': 'UNRECOGNIZED_FIELD'}, 'message': "Unrecognized field in the query: 'ad_group.explorer_auto_optimizer_setting.opt_in'."}], 'requestId': 's8EJceWZuYmQ1QZcy7mk-w'}]}}:
tap_googleads/client.py
2.listaccesiblecustomers doesn't exist
Incremental Data
429 RESOURCE_EXHAUSTED - Should be retried
We are not retrying 429's we should at least retry them. Google says to wait 15 minutes but maybe our standard retry logic will just work? Let's try it
Make login-customer-id optional
With the customer hierarchy this can be determined dynamically
Cleanup
Is no longer valid.
- Remove TODOs
_constant for geolocation - Remove country filter, don't query for every customer
accesible_customers no primary key
raise Exception("key_properties field is required")
Auto add _sdc_primary_key to schema
Each schema file needs to add the _sdc_primary_key field. We could auto add this, or at least write a test to be sure this was added to the schema for the stream. Not a huge deal, but a nice quality of life thing!
Map out other reports needed
https://cloud.google.com/bigquery-transfer/docs/adwords-transformation Looks interesting
primary_key incorrect - causes failures in target-postgres, and target-snowflake (probably more)
As the tap stands there are a couple of issues when using target-postgres, both variant Meltano and Transferwise.
1. Using target-postgres causing errors.
Meltano variant of target-postgres results in a psycopg2 error of UndefinedColumn, as it unpackages the json objects with different names to the expected.
Transferwise variant errors out with key_properties field is required.
The work around in my use case was to remove primary_keys all together (to have it work with multiple target-postgres variants), but realise that isn't a great solution, and not suitable in general.
target-postgres Transferwise error
target-postgres | File "/home/dw/Documents/hahaha/tap-googleads/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/__init__.py", line 209, in persist_lines
target-postgres | raise Exception("key_properties field is required")
target-postgres | Exception: key_properties field is required
meltano | Loading failed (1): Exception: key_properties field is required
target-postgres Meltano error
target-postgres | return super(DictCursor, self).execute(query, vars)
target-postgres | psycopg2.errors.UndefinedColumn: column "customer_client.id" named in key does not exist
target-postgres | LINE 1: ..., "customer_client__time_zone" character varying, PRIMARY KE...
target-postgres | ^
target-postgres |
meltano | Loading failed (1): (see above)
meltano | ELT could not be completed: Loader failed
2. If you do delete the primary keys, then use the Meltano variant of target-postgres you only get your last synced customer data.
Using the Meltano variant seems to make it so each time a customer's data is looped and added to the postgres table, that the table is just completely re-created, so you only ever get the last customers data. Also when the tap loops through the customers, all the postgres tables are created over and over for each customer.
I didn't end up looking too much into this issue, and it could be a more of a loader issue rather than tap, but I've never run into the loader behaving like this before so thought I mention it.
For us, we use the Meltano variant of target-postgres for some other reasons by default, so my work around was just to sync a single customer, no looping, no overwriting. Again more a fix for our use case than in general.
Would be awesome to hear some thoughts/ideas on these issues, not sure if anyone has seen a target-postgres creating tables over and over? I'm guessing its some quirk with the child streams?
JSON Schema linter
https://github.com/AutoIDM/tap-googleads/pull/28/files
PR has a bunch of linting fixes for JSON Schema data, it'd be nice if we could just codify this and then not question what the format should be.
Add Conversions by Location Stream
SELECT campaign_criterion.location.geo_target_constant, campaign.name, campaign_criterion.bid_modifier, segments.date, segments.conversion_action_category, metrics.conversions FROM location_view WHERE segments.date DURING LAST_7_DAYS AND campaign_criterion.status != 'REMOVED'
Custom Reports
Allow custom reports via a configuration option. Similar to https://gitlab.com/autoidm/autoidm-tap-bamboohr/-/blob/master/meltano.yml#L20
Add a license
Was looking through my forks and happened to notice there wasn't a license.
Upgrade SDK for new features!
- Flattening and some other nice features are there. Flattening would be nice
Implement State
This is a new feature request. It would be good to be able to control the number of days pulled from Google Ads in certain taps. Most of time this is controlled in the Google Ads query with something like WHERE segments.date DURING LAST_7_DAYS. Changing to a segments.date BETWEEN '2019-01-01' AND '2019-01-31' strategy controlled by the configuration parameters start_date and end_date would be ideal.
State file contains duplicate entries for partition
ValueError: State file contains duplicate entries for partition: {state_partition_context}. cmd_type=extractor job_id=googleads2jsonl_3 name=tap-googleads run_id=2fc9a784-47be-4217-99a5-13318a70c330 stdio=stderr 2022-03-22T19:39:15.932107Z [info ] Matching state values were:
To replicate
- Run
meltano elt tap-googleads target-jsonl --job_id=abcd - Run
meltano elt tap-googleads target-jsonl --job_id=abcd
You'll see the failure on run 2.
Roadmap
- CI
- POC (0.0.1)
- POC (0.1.1) - 5 more streams
- 1.0.0 Release - All Streams, with Discovery and everything
- Rate Limiting
- Error Handling
- Testing with actual data
- Be sure documentation is clear (Should happen along the way)
- Add MIT License
- Release 1.0.0 to PyPi
- Submit to Meltano Hub
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.