I tried to reproduce the example with amazon MSK and my S3 but failed. Debug log:<

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value about kafka-delta-ingest HOT 3 OPEN

delta-io commented on June 7, 2024 1

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value

from kafka-delta-ingest.

Comments (3)

rtyler commented on June 7, 2024

This looks like something caused kafka-delta-ingest to fail to configure credentials for interacting with S3. I'm wondering if you could share a bit more about the runtime environment (EC2, Fargate, etc?) and what form of credentials are being configured for kafka-delta-ingest

from kafka-delta-ingest.

huyhandes commented on June 7, 2024

This looks like something caused kafka-delta-ingest to fail to configure credentials for interacting with S3. I'm wondering if you could share a bit more about the runtime environment (EC2, Fargate, etc?) and what form of credentials are being configured for kafka-delta-ingest

This is how I ran the example:

export AWS_ENDPOINT_URL=https://s3.ap-southeast-1.amazonaws.com
export AWS_REGION=ap-southeast-1
export AWS_ACCESS_KEY_ID=OUR_KEY
export AWS_SECRET_ACCESS_KEY=OUR_ACCESS_SECRET

docker run -it --rm --network=host \
  -e RUST_LOG="debug" \
  -e AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID} \
  -e AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY} \
  -e AWS_ENDPOINT_URL={AWS_ENDPOINT_URL} \
  -e AWS_REGION={AWS_REGION} \
  -e RUST_BACKTRACE=full \
  kdi:0.1 \
  ingest deltalake-test-ingest s3a://BUCKET/deltalake-test-ingest \
  --allowed_latency 60 \
  --kafka "OUR_KAFKA_1:9093,OUR_KAFKA_2:9093" \
  -K socket.keepalive.enable=true \
  -K metadata.max.age.ms=180000 \
  -K heartbeat.interval.ms=3000 \
  -K session.timeout.ms=30000 \
  -K debug=broker,security,protocol \
  --app_id tester-mode-on \
  --transform "meta.kafka.topic: kafka.topic" \
  --auto_offset_reset earliest

from kafka-delta-ingest.

rtyler commented on June 7, 2024

@huyhandes would you be able to try this again with the latest main? We have some refactorings and crate upgrades that have landed this past weekend which might just resolve this issue 🤔

from kafka-delta-ingest.

Recommend Projects