databricks / benchmarks Goto Github PK
View Code? Open in Web Editor NEWA place in which we publish scripts for reproducible benchmarks.
License: Other
benchmarks's People
Contributors
Stargazers
Watchers
Forkers
ciso-drew tantonescu zhensongqian oeegee windpiger gengliangwang fanwu72 harlixxy sudheer0553 xfyecn data-mining minhqnguyen miguelperalvo yaominzh wangshuo128 ngupta23 kartikmehta09 guoyang2011 ns1123 mengjin001 iftachsc ankitsharma1990 shawnzhong thanhta2010 varuy322 vingao haochending whhsu2 arpitgrover87 llenroc panbingkun ideallain mxdzs0612benchmarks's Issues
Configuration files for Presto missing
Please publish configs for Presto as you did for Spark.
Spark Standalone Mode - production
When you test benchmark in standalone mode, pandas vs pyspark.
I got curios, can it implement in production ?
How you set how many slave?
Executor cores?
Executor memory?
Executor number?
And if I used spark on docker will it be better ? or best performance when on baremetal spark ?
About pre-generated TPC-DS data
Would databricks provides public S3 bucket containing pre-generated TPC-DS data?
Thanks.
This benchmark is invalid
More specifically:
All sounds fair, at least for Flink and Spark. We will note that the fact that Spark and Flink use in-memory data generation and KStreams goes through Kafka is itself a bias against KStreams.
`kafka/bin/kafka-topics.sh: No such file or directory` when running KStreamsYahooRunner
We executed the notebook on a Databricks cluster with version 6.2 (includes Apache Spark 2.4.4, Scala 2.11), and ran into the following problem on command 10:
Wrote 162 bytes.
executing command - kafka/bin/kafka-topics.sh --delete --topic output --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
bash: kafka/bin/kafka-topics.sh: No such file or directory
FAILED: command - kafka/bin/kafka-topics.sh --delete --topic output --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
executing command - kafka/bin/kafka-topics.sh --create --topic output --partitions 1 --replication-factor 1 --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
bash: kafka/bin/kafka-topics.sh: No such file or directory
FAILED: command - kafka/bin/kafka-topics.sh --create --topic output --partitions 1 --replication-factor 1 --zookeeper 1212-215524-fry930-10-172-248-143:2181 on host: 1212-215524-fry930-10-172-248-143
java.lang.RuntimeException: Command failedIt seems that the system cannot find the file kafka/bin/kafka-topics.sh for some reason.
Is there anything we can do to fix this? Thanks in advance.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.