kgpayne / tap-singer-jsonl Goto Github PK

Support common start_date setting, being the earliest record date to sync.

This effectively means:

• Ordering discovered files by timestamp (per stream)
• Fetching the first file timestamped before the start_date (in case it contains records that fall on either side of the threshold).
• Filtering all records >= start_date

Incremental refresh by means of state tracking.

This effectively means:

• Ordering discovered files (per stream) by timestamp.
• Emitting the timestamp of the last read file (per stream) as STATE.
• Filtering discovered files (per stream) according to received timestamp on subsequent runs.

Accept input catalog, apply metadata and selection rules.

This would effectively mean:

• Discovering all available streams in configured files/folders/s3 paths, and filtering at the stream level.
• Applying selection criteria to each record as it is read from file.
  • We already inject _sdc_* metadata, so this isn't such a big deal.