Comments (2)
Matthieu68857
commented on August 16, 2024
We do have the same errors on our side:
- We only have one task per connector and table
- No other DML by anything outside the connector
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"location" : "q",
"locationType" : "parameter",
"message" : "Could not serialize access to table xxx due to concurrent update",
"reason" : "invalidQuery"
} ],
"message" : "Could not serialize access to table xxx due to concurrent update",
"status" : "INVALID_ARGUMENT"
}
from kafka-connect-bigquery.
jjaimon
commented on August 16, 2024
I made following changes to reduce the errors.
tasks.max: "1",
intermediateTableSuffix: mysuffix,
bufferSize: 100000
maxWriteSize: 10000
tableWriteSize: 1000
The above changes are in the connection configuration. I think the first three changes helped us reduce the errors to nearly 0.
from kafka-connect-bigquery.
Related Issues (20)
- Provide support to authenticate without service account keys
- Type Numberic(10) from Postgres not supported. Should be NUMERIC HOT 2
- Merge flush tmp tables left unflushed
- GCSToBQLoadRunnable does not detect error during load and removes blobs even though they were not loaded
- GCSToBQLoadRunnable doesn't respect GCS folder
- setting different `clusteringPartitionFieldNames` across different tables in the same connector
- schemaless ( JSON events ) to bq fail to create table HOT 2
- Version 2.4.4 can't be found
- Support for Storage Write API HOT 18
- Support for partition filter required config when using UPSERT
- Convert io.debezium.time.MicroTimestamp to Bigquery TIMESTAMP HOT 2
- Timestamp conversion issue
- ERROR : INVALID_ARGUMENT jobInternalError HOT 2
- Setting kafkaKeyFieldName, when keys are string and topic has no schema, is not working HOT 2
- Infinite option for bigQueryRetry HOT 3
- Connector Not Restarting after BigQueryConnectException HOT 3
- Separate billing project & destination project HOT 2
- Records Duplication in BigQuery when Streaming Data from Kafka Topic via kafka-connect-bigquery HOT 8
- JSON data type is not working HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-bigquery.