schemaless ( JSON events ) to bq fail to create table about kafka-connect-bigquery HOT 2 OPEN

@raphaelauv , connector does not support table creation and schema update on JSON input. If you create the table manually with appropriate schema and send payload of type Map, data ingestion should not fail.

from kafka-connect-bigquery.

has anyone found a way to solve this?
the only thing i could get working is to use a StringConverter and the HoistField SMT to wrap the whole thing in a single value. but the table just has a single column with the entire string as a string. would love a way to have the schema infered from the json and flatten it.

{
"name": "BigQuerySinkConnector-metadata-5",
"config": {
"connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
"tasks.max": "1",
"topics.regex": ".*\.json$",
"project": "montu-bi-playground",
"defaultDataset": "gh_himashan",
"keyfile": "...",
"keySource": "JSON",
"autoCreateTables": "true",
"autoUpdateSchemas": "true",
"allowNewBigQueryFields": "true",
"allowBigQueryRequiredFieldRelaxation": "true",
"sanitizeTopics": "true",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter.schemas.enable": "false",
"errors.tolerance": "all",
"errors.deadletterqueue.topic.name": "bqSinkTopicDlq",
"errors.deadletterqueue.topic.replication.factor": "1",
"errors.deadletterqueue.context.headers.enable": "true",
"transforms": "WrapAsStruct",
"transforms.WrapAsStruct.type": "org.apache.kafka.connect.transforms.HoistField$Value",
"transforms.WrapAsStruct.field": "message_value"
}
}

from kafka-connect-bigquery.