ucalgary / bottledwater-pg Goto Github PK

KIP-32 added a timestamp field to the Kafka message format. Bottled Water should produce messages with a timestamp, either from the wall time or based on an attribute in the table being streamed.

When streaming from a Converis database, specifying a wildcard pattern for the --tables option fails for actionlog*. Bottled Water reports no matching tables. It should match actionlogrecord and actionlogitem.

Specifying attributevalues_* does work, though.

Update the various Makefiles and the Travis config to use the new Dockerfiles and Docker Compose file.

The new Bottled Water client image is missing the original image's entrypoint script. It also lacks instructions for ENTRYPOINT and CMD.

The bottledwater-docker-wrapper.sh entrypoint script sets the --schema-registry option if it is able to find a host entry for schema-registry. This makes it impossible to use the json output format if a Schema Registry service is reachable, because Bottled Water exits with the following message.

Specifying --schema-registry doesn't make sense for --output-format=json

Bottled Water generates Kafka message keys as a dict based on the table row primary key. This is inconvenient when using the resulting stream with some Kafka Streams operations, because dict comparisons for the same key can come out as false.

Provide an option for emitting the value of the primary key as the message key, instead of wrapping it in a dict. Here's an example, expressed in JSON.

Current message key:

{
  "id": {
    "long": 12345
  }
}

Flat message key:

12345

Bottled Water outputs datetimes as a dict of datetime components like the following.

"created": {
  "DateTime": {
    "year": "2017",
    "month": "03",
    "date": "14",
    "hour": "15",
    "minute": "42",
    "second": "49",
    "micro": "230990"
  }
}

It would be handy to have an option to output an ISO 8601 combined date and time format instead.

2017-03-14T15:42:49.230990

The json output format in Bottled Water is a literal json representation of an Avro formatted message. This can be highly inconvenient to parse. Here is an example of the current output.

{
  "id": {
    "long": 3688267
  },
  "language": {
    "string": "en_US"
  },
  "created": {
    "DateTime": {
      "year": "2017",
      "month": "03",
      "date": "14",
      "hour": "15",
      "minute": "42",
      "second": "49",
      "micro": "230990"
    }
  }
}

In the example, the value of each attribute (id, language, and created) is a dict with one item whose key is the value type, and value is the actual value. This requires readers to either know ahead of time what the value type of each attribute is in order to get the underlying value, or infer the value key by iterating or searching over the dict.

It would be convenient if there is an option to directly emit the value of each attribute, like the following.

{
  "id": 3688267,
  "language": "en_US",
  "created": {
    "year": "2017",
    "month": "03",
    "date": "14",
    "hour": "15",
    "minute": "42",
    "second": "49",
    "micro": "230990"
  }
}

Or, if it is desirable to preserve the type information, organize it as a list/array/tuple, so that it is easy to get the type and underlying value without needing fore-knowledge or potentially expensive operations over dict.

{
  "id": ["long", 3688267],
  "language": ["string", "en_US"],
  "created": ["DateTime", {
    "year": "2017",
    "month": "03",
    "date": "14",
    "hour": "15",
    "minute": "42",
    "second": "49",
    "micro": "230990"
  }]
}