dataops-tk / tapdance Goto Github PK

Passing in a comma-separated list of tables via tap_settings[“TABLE_NAME”] should cause tapdance to sync only the tables specified. However, the current behavior is that this filter gets ignored and it instead uses the default value of “*”, which syncs all the tables.

I suspect the issue is on line 111 in syncs.py which currently checks the table_name variable first before looking at tap_settings[“TABLE_NAME”]. If they’re both populated (with a truthy value), the OR condition will return the value from table_name.

Given the rule:

!x*

Expected result is to exclude x$somethingorother but not exclude myextratable. Currently both are getting excluded.

Most likely this bug is caused by the table matching logic not properly checking for left-side boundary when performing the regex check.

The workflow cicd.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

An inquiry from slack came through regarding whether this can be run without any python wrapper.

This would be good to explore and document.

From the logs, here's the docker run statement from plan when running locally. (The plan statement is Tapdance's replacement for discover.)

docker run  -v C:\Files\Source\tapdance\samples\taps:/projects/my-project -w /projects/my-project dataopstk/tapdance:tap-covid-19 tapdance plan covid-19

And here's the sync command:

docker run -v C:\Files\Source\tapdance\samples\taps\covid-19:/projects/my-project -w /projects/my-project dataopstk/tapdance:covid-19-to-csv tapdance sync covid-19 csv

And a screenshot of a sync running natively in ECS (dockerized):

(Note that currently, running the sync command also runs plan.)

Esp.:

• Pardot Visits
• GL table

Expected default behavior:

• If missing primary-key but replication-key is okay:
  • incremental load (no upsert or duplicate detection)
• If missing primary-key and no replication-key exists:
  • full table, full replace (unless behavior overridden)

TODO