databricks / benchmarks Goto Github PK

We executed the notebook on a Databricks cluster with version 6.2 (includes Apache Spark 2.4.4, Scala 2.11), and ran into the following problem on command 10:

Wrote 162 bytes.
executing command - kafka/bin/kafka-topics.sh --delete --topic output --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
bash: kafka/bin/kafka-topics.sh: No such file or directory
FAILED: command - kafka/bin/kafka-topics.sh --delete --topic output --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
executing command - kafka/bin/kafka-topics.sh --create --topic output --partitions 1 --replication-factor 1 --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
bash: kafka/bin/kafka-topics.sh: No such file or directory
FAILED: command - kafka/bin/kafka-topics.sh --create --topic output --partitions 1 --replication-factor 1 --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
java.lang.RuntimeException: Command failed

It seems that the system cannot find the file kafka/bin/kafka-topics.sh for some reason.

Is there anything we can do to fix this? Thanks in advance.

Would databricks provides public S3 bucket containing pre-generated TPC-DS data?

Thanks.

When you test benchmark in standalone mode, pandas vs pyspark.
I got curios, can it implement in production ?

How you set how many slave?
Executor cores?
Executor memory?
Executor number?

And if I used spark on docker will it be better ? or best performance when on baremetal spark ?

Please publish configs for Presto as you did for Spark.

SEE https://data-artisans.com/blog/curious-case-broken-benchmark-revisiting-apache-flink-vs-databricks-runtime

More specifically:
All sounds fair, at least for Flink and Spark. We will note that the fact that Spark and Flink use in-memory data generation and KStreams goes through Kafka is itself a bias against KStreams.