tzachshabtay / krowser Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 1.0 1.5 MB

Web UI to browse kafka, schema registry and kafka-connect

License: MIT License

TypeScript 68.76% CSS 0.06% JavaScript 1.58% Dockerfile 0.76% HTML 0.28% Rust 28.56%

kafka kafka-connect schema-registry

krowser's People

Contributors

Stargazers

Watchers

Forkers

tanishq-dubey

krowser's Issues

Multi-topics search- autocomplete for selecting multiple topics

In the multi-topics search view, need to replace the simple select to be able to type the topics.
We can try material-ui autocomplete instead of the select (we do want to keep the select/clear all checkbox though).

Auto refresh

Have an auto-refresh toggle button for all the queries (thinking of a similar experience as kibana/grafana).

Show consumers per topic

We want to show the consumer groups which subscribe to a specific topic.
Waiting for support from kafkajs: tulios/kafkajs#755

UNKNOWN_TOPIC_OR_PARTITION

If kafka is restarted when krowser is running, krowser doesn't know how to recover, and has the following error:

Failed to fetch topic mytopic. Error: {"name":"KafkaJSNumberOfRetriesExceeded","retriable":false,"originalError":{"name":"KafkaJSProtocolError","retriable":true,"type":"UNKNOWN_TOPIC_OR_PARTITION","code":3},"retryCount":5,"retryTime":14262}

Edit mode

Currently krowser is read-only.
We can allow editing (under a feature flag, some people will probably just want a read-only experience) various things in kafka and schema-registry (like CRUD operations for topics, partition assignments, subjects, configs, modifying consumer group offsets).
Each one of those should be in its own feature flag for maximum flexibility.

Can't perform a React state update on an unmounted component

Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in Topics (created by Context.Consumer)


overrideMethod | @ | react_devtools_backend.js:2557
-- | -- | --
  | printWarning | @ | react-dom.development.js:88
  | error | @ | react-dom.development.js:60
  | warnAboutUpdateOnUnmountedFiberInDEV | @ | react-dom.development.js:23161
  | scheduleUpdateOnFiber | @ | react-dom.development.js:21169
  | enqueueForceUpdate | @ | react-dom.development.js:12678
  | Component.forceUpdate | @ | react.development.js:490
  | fetchTopic | @ | topics.tsx:120
  | async function (async) |   |  
  | fetchTopic | @ | topics.tsx:92
  | componentDidMount | @ | topics.tsx:87
  | async function (async) |   |  
  | componentDidMount | @ | topics.tsx:76
  | commitLifeCycles | @ | react-dom.development.js:19814
  | commitLayoutEffects | @ | react-dom.development.js:22803
  | callCallback | @ | react-dom.development.js:188
  | invokeGuardedCallbackDev | @ | react-dom.development.js:237
  | invokeGuardedCallback | @ | react-dom.development.js:292
  | commitRootImpl | @ | react-dom.development.js:22541
  | unstable_runWithPriority | @ | scheduler.development.js:653
  | runWithPriority$1 | @ | react-dom.development.js:11039
  | commitRoot | @ | react-dom.development.js:22381
  | finishSyncRender | @ | react-dom.development.js:21807
  | performSyncWorkOnRoot | @ | react-dom.development.js:21793
  | (anonymous) | @ | react-dom.development.js:11089
  | unstable_runWithPriority | @ | scheduler.development.js:653
  | runWithPriority$1 | @ | react-dom.development.js:11039
  | flushSyncCallbackQueueImpl | @ | react-dom.development.js:11084
  | flushSyncCallbackQueue | @ | react-dom.development.js:11072
  | scheduleUpdateOnFiber | @ | react-dom.development.js:21199
  | enqueueSetState | @ | react-dom.development.js:12639
  | Component.setState | @ | react.development.js:471
  | (anonymous) | @ | Router.js:34
  | listener | @ | history.js:155
  | (anonymous) | @ | history.js:173
  | notifyListeners | @ | history.js:172
  | setState | @ | history.js:288
  | (anonymous) | @ | history.js:311
  | confirmTransitionTo | @ | history.js:145
  | handlePop | @ | history.js:309
  | handlePopState | @ | history.js:294

This happens when navigating from the topics window when it's still loading (but can happen in other windows as well), need to add "is mounted" checks and return, or a cancellable promise: https://reactjs.org/blog/2015/12/16/ismounted-antipattern.html.

Custom decoders

Add the ability to extend krowser with custom decoders.

Schema destructure for union and array of objects

In the schemas view, we don't currently destructure union or arrays of objects into separate columns.

KSQL

Support KSQL view(s)

docker-compose example; confluent-kafka

Hi! I tried launch krowser with confluent kafka as broker.

My compose file:

version: '2.1'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.2.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'

  schema-registry:
    image: confluentinc/cp-schema-registry:6.2.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  krowser:
    image: tzachs/krowser:latest
    restart: unless-stopped
    hostname: krowser
    ports:
      - "9999:9999"
    environment:
      KAFKA_URL: broker:29092
      SCHEMA_REGISTRY_URL: http://localhost:8081

I confused with error:

krowser_1 | {"level":"ERROR","timestamp":"2021-07-05T20:32:25.000Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED 127.0.0.1:9092","retryCount":4,"retryTime":4080}

Could you explain what the right way to start work with confluent kafka as broker?

Best regards!

Uncaught (in promise) SyntaxError: Unexpected token

Uncaught (in promise) SyntaxError: Unexpected token < in JSON at position 0 async function (async) (anonymous) @ fetcher.tsx:410 callCallback @ react-dom.development.js:12490 commitUpdateQueue @ react-dom.development.js:12511 commitLifeCycles @ react-dom.development.js:19858 commitLayoutEffects @ react-dom.development.js:22803 callCallback @ react-dom.development.js:188 invokeGuardedCallbackDev @ react-dom.development.js:237 invokeGuardedCallback @ react-dom.development.js:292 commitRootImpl @ react-dom.development.js:22541 unstable_runWithPriority @ scheduler.development.js:653 runWithPriority$1 @ react-dom.development.js:11039 commitRoot @ react-dom.development.js:22381 finishSyncRender @ react-dom.development.js:21807 performSyncWorkOnRoot @ react-dom.development.js:21793 (anonymous) @ react-dom.development.js:11089 unstable_runWithPriority @ scheduler.development.js:653 runWithPriority$1 @ react-dom.development.js:11039 flushSyncCallbackQueueImpl @ react-dom.development.js:11084 flushSyncCallbackQueue @ react-dom.development.js:11072 discreteUpdates$1 @ react-dom.development.js:21893 discreteUpdates @ react-dom.development.js:806 dispatchDiscreteEvent @ react-dom.development.js:4168

async function (async) |   |  
-- | -- | --
  | (anonymous) | @ | fetcher.tsx:410
  | callCallback | @ | react-dom.development.js:12490
  | commitUpdateQueue | @ | react-dom.development.js:12511
  | commitLifeCycles | @ | react-dom.development.js:19858
  | commitLayoutEffects | @ | react-dom.development.js:22803
  | callCallback | @ | react-dom.development.js:188
  | invokeGuardedCallbackDev | @ | react-dom.development.js:237
  | invokeGuardedCallback | @ | react-dom.development.js:292
  | commitRootImpl | @ | react-dom.development.js:22541
  | unstable_runWithPriority | @ | scheduler.development.js:653
  | runWithPriority$1 | @ | react-dom.development.js:11039
  | commitRoot | @ | react-dom.development.js:22381
  | finishSyncRender | @ | react-dom.development.js:21807
  | performSyncWorkOnRoot | @ | react-dom.development.js:21793
  | (anonymous) | @ | react-dom.development.js:11089
  | unstable_runWithPriority | @ | scheduler.development.js:653
  | runWithPriority$1 | @ | react-dom.development.js:11039
  | flushSyncCallbackQueueImpl | @ | react-dom.development.js:11084
  | flushSyncCallbackQueue | @ | react-dom.development.js:11072
  | discreteUpdates$1 | @ | react-dom.development.js:21893
  | discreteUpdates | @ | react-dom.development.js:806
  | dispatchDiscreteEvent | @ | react-dom.development.js:4168

This happened after receiving "bad gateway" from the server in

krowser/src/client/kafka/messages/fetcher.tsx

Line 302 in bf86ffb

const response = await fetch(

this is caught in

krowser/src/client/kafka/messages/fetcher.tsx

Line 412 in bf86ffb

} catch (error) {

but then doesn't show the error to the user and reject a promise to get an error in the console instead.

Put schema registry under a feature flag

Currently we force people to have schema registry. We should have a feature flag for it to disable all features schema registry related, for those who use kafka without it.

Time picker UI not supported on firefox

We're currently using datetime-local for our datetime pickers which is not supported on firefox.
Material-ui are building a datetime picker we should use instead once it's hits stable: https://material-ui.com/components/pickers/

Configurable timeouts

Allow configurable timeouts for the various server requests.

SharedArrayBuffer will require cross-origin isolation as of M91, around May 2021

warning in browser, solution seems to be upgrading to react 17.0.2: https://stackoverflow.com/questions/66489286/sharedarraybuffer-will-require-cross-origin-isolation-as-of-m91-around-may-2021

Received NaN for the `value` attribute

This is shown in console every time we clear the limit or offset text fields

react_devtools_backend.js:2557 Warning: Received NaN for the `value` attribute. If this is expected, cast the value to a string.
    in input (created by ForwardRef(InputBase))
    in div (created by ForwardRef(InputBase))
    in ForwardRef(InputBase) (created by WithStyles(ForwardRef(InputBase)))
    in WithStyles(ForwardRef(InputBase)) (created by ForwardRef(Input))
    in ForwardRef(Input) (created by WithStyles(ForwardRef(Input)))
    in WithStyles(ForwardRef(Input)) (created by ForwardRef(TextField))
    in div (created by ForwardRef(FormControl))
    in ForwardRef(FormControl) (created by WithStyles(ForwardRef(FormControl)))
    in WithStyles(ForwardRef(FormControl)) (created by ForwardRef(TextField))
    in ForwardRef(TextField) (created by WithStyles(ForwardRef(TextField)))
    in WithStyles(ForwardRef(TextField)) (created by InputField)
    in InputField (created by Fetcher)
    in div (created by Fetcher)
    in div (created by ForwardRef(Toolbar))
    in ForwardRef(Toolbar) (created by WithStyles(ForwardRef(Toolbar)))
    in WithStyles(ForwardRef(Toolbar)) (created by Fetcher)
    in Fetcher (created by SingleTopicInput)
    in SingleTopicInput (created by Messages)
    in Messages (created by Context.Consumer)
    in Route (created by App)
    in div (created by App)
    in Router (created by BrowserRouter)
    in BrowserRouter (created by App)
    in ThemeProvider
    in Unknown (created by App)
    in App

Search by time range

In the 2 message search views (for a single topic and for multiple topics) we want to be able to search by time range (as an alternative to searching by offsets).
Waiting for support from kafkajs: tulios/kafkajs#604 or tulios/kafkajs#525

ag-grid enterprise

We currently use the open source ag-grid community version.
The enterprise version has more features which we can benefit from (enum filters, column selectionn, groupings and aggregations, and the ability to expand a row so we can show the json view inside the grid).

Need to check if ag-grid has a license which allows open source projects to use the enterprise edition. Otherwise, perhaps we can add it as a feature flag (the user will need to enter their ag-grid license key which will unlock the extra features).

Show last offset as of X time

Feature request:

Show the "last offset" as of a timestamp. Current method requires start/end time and loads everything inbetween those times - have to incrementally move the window to find "last as of X" with this current state and loading everything can be unpredictable.