The objective of this project is to test Audit Logs (audit topic & log files) the authorization of topics and the behavior of a client when being authorized / not authorized by the broker.

Note: By default the audit information authorizationInfo is out to an internal topic. This behaviour is not suitable if you want to use something like metricbeat to index this information.

This project has a dependency on a running Elastic stack We use the network created in this project to send data from filebeat to Elastic

$ git clone [email protected]:osodevops/docker-elk.git
$ cd docker-elk
$ docker-compose up --build

The environment deploys 1 Zookeeper and 1 Kafka broker, to start this example you can run the following:

To start the environment run:

docker-compose up -d

To stop the environment use (This will delete all volumes to avoid stale ACL configuration in Zookeeper):

docker-compose down -v

RBAC is disabled to use ACL only:

  broker:
    environment:
      KAFKA_AUTHORIZER_CLASS_NAME: io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
      KAFKA_CONFLUENT_AUTHORIZER_ACCESS_RULE_PROVIDERS: ZK_ACL

This test has following override configuration:

  broker:
    environment:
      KAFKA_CONFLUENT_SECURITY_EVENT_ROUTER_CONFIG: "{\"routes\":{\"crn:///kafka=*/group=*\":{\"consume\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"}},\"crn:///kafka=*/topic=*\":{\"produce\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"},\"consume\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"}}},\"destinations\":{\"topics\":{\"confluent-audit-log-events\":{\"retention_ms\":7776000000}}},\"default_topics\":{\"allowed\":\"confluent-audit-log-events\",\"denied\":\"confluent-audit-log-events\"},\"excluded_principals\":[\"User:kafka\",\"User:ANONYMOUS\"]}"

which is:

{
  "routes": {
    "crn:///kafka=*/group=*": {
      "consume": {
        "allowed": "confluent-audit-log-events",
        "denied": "confluent-audit-log-events"
      }
    },
    "crn:///kafka=*/topic=*": {
      "produce": {
        "allowed": "confluent-audit-log-events",
        "denied": "confluent-audit-log-events"
      },
      "consume": {
        "allowed": "confluent-audit-log-events",
        "denied": "confluent-audit-log-events"
      }
    }
  },
  "destinations": {
    "topics": {
      "confluent-audit-log-events": {
        "retention_ms": 7776000000
      }
    }
  },
  "default_topics": {
    "allowed": "confluent-audit-log-events",
    "denied": "confluent-audit-log-events"
  },
  "excluded_principals": [
    "User:broker",
    "User:ANONYMOUS"
  ]
}

The Kafka broker is configured with SASL/Plaintext authentication and Zookeeper is configured with SASL/Digest authentication. The brokers are adding ACLs to the Zookeeper nodes when they write and the broker will not allow access to topics without ACLs set (allow.everyone.if.no.acl.found=false).

This is a strict setting and the broker has 3 users configured

• admin - configured as super user
• broker - configured as super user
• producer
• consumer

The script create_topic creates the topic first-topic and set the ACLs for user producer to write and the user consumer to read from this topic (in the case of the consumer, from any consumer group id).

The JAAS files used to configure the usernames and passwords, as well as the client credentials used by the broker are in the directory security of the repo (and they are mounted as /opt/security in broker and in zookeeper).

The broker will expose the port 9092 as a SASL authenticated port in the localhost.

This testing environment docker-compose.yml is using Confluent docker images 5.5.1 (This may work with older versions)

log4j.properties.template is appeneded to the default configuration which creates a authorizerAppender to log authorization events to /var/log/kafka/kafka-authorizer.log

It is required to have Kafka CLI tool installed to be able to use this environment. The best way to do it in a Mac is:

brew install kafka

Run the ./create_topic.sh script to connect to broker and create the topic with the included ACLs.

You are now able to produce messages to the topic by running:

kafka-console-producer --broker-list localhost:9092 --producer.config client-properties/producer.properties --topic first-topic

The command above assumes that the topic first-topic was created and the ACLs for producing were assigned. To perform this action just run the script create_topic.sh

Similarly to consumer from the topic:

kafka-console-consumer --bootstrap-server localhost:9092 --consumer.config client-properties/consumer.properties --group test-consumer-group --topic first-topic

The command above assumes that the topic first-topic was created and the ACLs for producing were assigned. To perform this action just run the script create_topic.sh

To best understand what audit information is logged we have created the following scenarios for you to test on your own.

The commands below are executed from the directory where this repo was cloned (due to the client-properties relative directory)

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can read more about kafkacat here https://github.com/edenhill/kafkacat

$ brew install kafkacat
######### FIND LEAD KAFKA BROKER ######### 
$ kafkacat -L -b localhost:9092 \
-X security.protocol=sasl_plaintext \
-X sasl.mechanisms=PLAIN -X sasl.username=admin -X sasl.password='admin-secret'

TODO!!