Follow up from #1 and #4, I have noticed a problem with the JSON output produced by eventbridge and kinesis modules, JSON output could exceed the maxRecordSize value because the encoding is done without the validation.

A potential alternative for this is to set a featureFlag that defines whether the output will be encoded as TSV or JSON, this way, we could leverage this approach for all outputs, similarly, we could define another featureFlag that specifies whether the JSON output is flattened or not (or use a single flag that specifies the format).

Its ideal to cover this important functionality with tests.

See Enrich#serializeEnriched

We tried the release workflow but got issues while publishing images to the docker registry (https://github.com/opensnowcat/opensnowcat-enricher/actions/runs/7487179270/job/20379062445), which makes sense because no account has been linked yet.

Let's create the docker account and configure the repository to use it for publishing the container images.

When executing the release workflow, the jar files must be attached to the github release.

We have tried running this workflow (https://github.com/opensnowcat/opensnowcat-enricher/actions/runs/7487179270/job/20379062445) and got a few warnings saying that the jar files can't be found:

🤔 Pattern 'modules/pubsub/target/scala-2.12/opensnowcat-enrich-pubsub-v0.0.1.jar' does not match any files.
🤔 Pattern 'modules/kinesis/target/scala-2.12/opensnowcat-enrich-kinesis-v0.0.1.jar' does not match any files.
🤔 Pattern 'modules/kafka/target/scala-2.12/opensnowcat-enrich-kafka-v0.0.1.jar' does not match any files.
🤔 Pattern 'modules/nsq/target/scala-2.12/opensnowcat-enrich-nsq-v0.0.1.jar' does not match any files.
🤔 modules/pubsub/target/scala-2.12/opensnowcat-enrich-pubsub-v0.0.1.jar,modules/kinesis/target/scala-2.12/opensnowcat-enrich-kinesis-v0.0.1.jar,modules/kafka/target/scala-2.12/opensnowcat-enrich-kafka-v0.0.1.jar,modules/nsq/target/scala-2.12/opensnowcat-enrich-nsq-v0.0.1.jar not include valid file.

This is likely caused by our partial renames introduced at #5 .

While the workflow succeeded, the github release does not have any jars attached.

Follow up from #3, #4 introduced a few important test cases that need more work in order to enable them (EnrichEventbridgeSpec.scala), let's fix and enable the tests.

Hint: It seems that the problem is that we are encoding the json with flattened fields which can't be parsed directly as an Event, we should find the way to do this.

We tried the release workflow and the maven central step failed (https://github.com/opensnowcat/opensnowcat-enricher/actions/runs/7487179270/job/20379062132), which is expected because we have not setup the account yet.

In order to do this, first we need to setup a sonatype account, which is explained by https://central.sonatype.org/publish/publish-guide/

Depends on #12.

Add a Eventbridge module

What is expected?

Data sent directly to EventBridge

Follow up from #5, we need to get rid of com.snowplowanalytics in anything that's released.

Github templates are affected too: https://github.com/opensnowcat/opensnowcat-enricher/tree/develop/.github

We tried running the workflow (see https://github.com/opensnowcat/opensnowcat-enricher/actions/runs/7487073724/job/20378732392?pr=7) but it failed with this error: ERROR: Authentication details are not configured.

Everything indicates that we need to create an account, then, fill the necessary secrets in github secrets:

• https://docs.lacework.net/onboarding/integrate-the-lacework-inline-scanner-with-ci-pipelines#configuration-using-environment-variables

Let's fix this workflow.

On the Kinesis module we want to add support for a new “format” for GOOD rows, a flattened JSON.

What is expected?

A JSON flattened in Kinesis, ready to be leveraged by other AWS services.

It is ideal to keep our dependencies up to date.

Hi, first of all thanks for your efforts in providing this repo and new features

I deployed 1.1.0 and using config

{
  ...
  "experimental": {
    "customOutputFormat": {
      "type": "FlattenedJson"
    }
  }
  ...
}

I am using pubsub bigquery subscription but the writing to bq does not work because of date/time/timestamp need to be integer in the JSON according to https://cloud.google.com/pubsub/docs/bigquery#date_time_int

could we plan a fix here. I am also open to contribute/open a PR if needed