artie-labs / transfer Goto Github PK

DynamoStream exports have a marshalled JSON structure that may match other sources too

As summed up from this PR: #94

Move the lock from the database to the table.

When doing so, add in race detection.

Hello !

Thanks for this amazing project !

Do you plan to add PostgreSQL support as a destination?

Hi!
Thanks for great project.

Would you like to add support for ClickHouse?
https://clickhouse.com/docs/en/

CollapsingMergeTree could be good table engine for stream replication transaction logs

Hi,I have another data warehouse not in transfer supported destination list. My data warehouse provide jdbc drive and python connector but without a go driver. In this case, How to develop a new transfer client for this new destination data warehouse?

Inspired by: https://news.ycombinator.com/item?id=36848483

Search and reading experience is significantly better.

Gitbooks also has CI and previews built-in.

Muy buenos dias,
Como se puede trabajar Artie con bases de datos Microsoft SQL Server?

I want to build a docker image in the github actions pipeline

Tombstone messages has no data in the body, let's filter it out.

With this PR #27, Postgres will support schemas.

We should have feature parity for MongoDB so we avoid the same issues listed in the PR.

https://cloud.google.com/bigquery/docs/write-api

Context https://artie-technologies.slack.com/archives/C04Q7GG6DT9/p1693997817550579

Right now, MERGE code is all independent.

There's quite a bit of bloat by having to copy similar LOC across each destination.

• Ability to specify append-only
• Ability to specify parquet

Hi,I have several kilometers tables to be sync from Postgres to snowflake, I want config a rule to map those n
source tables to 1 topic to n destination tables,followed by this guideline
https://docs.artie.so/tutorials/debezium-topic-reroute-smt ,
1,in a docket compose,which config file should I put parameters about # Re-route them all to the same topic?
2,how to configure reroute mapping rule such as:
n source tables to 1 topic to n destination tables

n source tables to 1 topic to 1 destination table

1 source tables to 1 topic to n destination tables

BR

We are relying on the Kafka message timestamp to emit ingestion lag.

Whenever there is a broker failover, the timestamp gets out of sync for a few minutes.

Such that we can monitor and observe the Kafka topic lag

Is there a plan to add support for extracting CDC from Oracle databases (as the source database) anytime soon?

As per title. Support spans, traces and custom tags into context

2023-06-24 22:13:24 time="2023-06-24T14:13:24Z" level=warning msg="skipping message..." logrus_error="can not add field "topic"" error="cannot unmarshall key, key: , err: key is nil" key= offset=17 value="{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"default":0,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":true,"field":"column_added_0001"},{"type":"string","optional":true,"field":"column_added_0002"},{"type":"string","optional":true,"field":"column_added_0003"},{"type":"string","optional":true,"field":"column_added_0004\

Today, we only have Kafka ingestion lag.

How about the total lag?

From: https://docs.snowflake.com/en/sql-reference/intro-summary-data-types.html

We are currently only supporting TIMESTAMP which strips the TZ locale from the value.

Line 39 - "To see the current supported databases" should be "To see the currently supported databases"

v. 2.0.58

Transfer generates BigQuery temp tables with suffixes that include capital letters, e.g. my_table___artie_dM9YWyeakM.

However, those suffixes are downcased in the drop statements, e.g. my_table___artie_dm9ywyeakm. This has the effect that the temporary table is not dropped at all, as it is does not exist. Lots of temp tables accrete in the dataset until they expire several hours later.

Streaming Insert has the limitation of 10MB, merge doesn't. If we do get hit by the 10Mb limit, just iterate over the 10 Mb batch

Full context

Check the in-memory DB and if the size exceeds X, then flush.

Effectively, the equation becomes, whichever one is sooner:

• Flush interval (10s)
• Numbers of rows within the in-memory DB (15k)
• Size of the in-memory DB

BigQuery CDC: https://cloud.google.com/blog/products/data-analytics/bigquery-gains-change-data-capture-functionality

Remove this need so we can loosen the requirements that we have around partition keys. In the future, we can explore workloads that have a "custom" primary key (e.g uniq constraint but not specified primary key).

We can get this from the retData from an event.

debezium/debezium#1717

So users can set it and forget it.

The fully qualified name for a table within a DB looks like this: schema.table.

Allow 2 following options to be added:

1. RegEx (schema.*)
2. Explicit "ignore" of tables.

AD will work if the table is found in (1) and not in (2). This is to maintain parity with Debezium. Artie and Debezium will only allow this search for non-system tables.