tpcds-hdinsight
Goal of this project is to help generate TPCDS data with hive and create your own HDInsight benchmarks for various engines
- Hive
- Interactive Hive(LLAP)
- Spark
- Presto
How to use with Hive CLI
-
Clone this repo.
git clone https://github.com/hdinsight/tpcds-hdinsight/ && cd tpcds-hdinsight
-
Run TPCDSDataGen.hql with settings.hql file and set the required config variables.
/usr/bin/hive -i settings.hql -f TPCDSDataGen.hql -hiveconf SCALE=10 -hiveconf PARTS=10 -hiveconf LOCATION=/HiveTPCDS/ -hiveconf TPCHBIN=resources
Here,
SCALEis a scale factor for TPCDS. Scale factor 10 roughly generates 10 GB data, Scale factor 1000 generates 1 TB of data and so on.PARTSis a number of task to use for datagen (parrellelization). This should be set to the same value asSCALE.LOCATIONis the directory where the data will be stored on HDFS.TPCHBINis where the resources are found. You can specify specific settings in settings.hql file. -
Now you can create tables on the generated data.
/usr/bin/hive -i settings.hql -f ddl/createAllExternalTables.hql -hiveconf LOCATION=/HiveTPCDS/ -hiveconf DBNAME=tpcds
Generate ORC tables and analyze
hive -i settings.hql -f ddl/createAllORCTables.hql -hiveconf ORCDBNAME=tpcds_orc -hiveconf SOURCE=tpcds hive -i settings.hql -f ddl/analyze.hql -hiveconf ORCDBNAME=tpcds_orc
-
Run the queries !
/usr/bin/hive -database tpcds_orc -i settings.hql -f queries/query12.sql
How to use with Beeline CLI
-
Clone this repo.
git clone https://github.com/hdinsight/tpcds-hdinsight && cd tpcds-hdinsight
-
Upload the resources to DFS.
hdfs dfs -copyFromLocal resources /tmp
-
Run TPCDSDataGen.hql with settings.hql file and set the required config variables.
beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f TPCDSDataGen.hql -hiveconf SCALE=10 -hiveconf PARTS=10 -hiveconf LOCATION=/HiveTPCDS/ -hiveconf TPCHBIN=`grep -A 1 "fs.defaultFS" /etc/hadoop/conf/core-site.xml | grep -o "wasb[^<]*"`/tmp/resources
Here,SCALEis a scale factor for TPCDS. Scale factor 10 roughly generates 10 GB data, Scale factor 1000 generates 1 TB of data and so on.PARTSis a number of task to use for datagen (parrellelization). This should be set to the same value asSCALE.LOCATIONis the directory where the data will be stored on HDFS.TPCHBINis where the resources are found. You can specify specific settings in settings.hql file. -
Now you can create tables on the generated data.
beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/createAllExternalTables.hql -hiveconf LOCATION=/HiveTPCDS/ -hiveconf DBNAME=tpcds
Generate ORC tables and analyze
beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/createAllORCTables.hql -hiveconf ORCDBNAME=tpcds_orc -hiveconf SOURCE=tpcds beeline -u "jdbc:hive2://`hostname -f`:10001/;transportMode=http" -n "" -p "" -i settings.hql -f ddl/analyze.hql -hiveconf ORCDBNAME=tpcds_orc
-
Run the queries !
beeline -u "jdbc:hive2://`hostname -f`:10001/tpcds_orc;transportMode=http" -n "" -p "" -i settings.hql -f queries/query12.sql
If you want to run all the queries 10 times and measure the times it takes, you can use the following command:
for f in queries/*.sql; do for i in {1..10} ; do STARTTIME="`date +%s`"; beeline -u "jdbc:hive2://`hostname -f`:10001/tpcds_orc;transportMode=http" -i settings.hql -f $f > $f.run_$i.out 2>&1 ; SUCCESS=$? ; ENDTIME="`date +%s`"; echo "$f,$i,$SUCCESS,$STARTTIME,$ENDTIME,$(($ENDTIME-$STARTTIME))" >> times_orc.csv; done; done;
FAQ
Does it work with scale factor 1?
No. The parrellel data generation assumes that scale > 1. If you are just starting out, I would suggest you start with 10 and then move to standard higher scale factors (100, 1000, 10000,..)
Do I have to specify PARTS=SCALE ?
Yes.
How do I avoid my session getting killed due to network errors while long running benchmark?
Use byobu. Type byobu which will start a new session and then run the command. It will be there when you come back even if your network connection is broken.
How do I generate partitioned text tables ?
After generating raw data(step 3a), use the following command:
hive -i settings.hql -f ddl/createAllTextTables.hql -hiveconf TEXTDBNAME=tpcds_text -hiveconf SOURCE=tpcds
This will generate tpcds_text database with all the tables in text format.
How do I generate Parquet data?
After generating raw data(step 3a), use the following command:
hive -i settings.hql -f ddl/createAllParquetTables.hql -hiveconf PARQUETDBNAME=tpcds_pqt -hiveconf SOURCE=tpcds
This will generate tpcds_pqt database with all the tables in parquet format.
How do I run the queries with Spark?
Spark thriftserver listens on 10002 instead of hive thrift server listening on 10001. So replace the connection url appropriately. For example, running the all the queries 10 times with Spark,
for f in queries/*.sql; do for i in {1..10} ; do STARTTIME="`date +%s`"; beeline -u "jdbc:hive2://`hostname -f`:10002/tpcds_orc;transportMode=http" -i sparksettings.hql -f $f > $f.run_$i.out 2>&1 ; SUCCESS=$? ; ENDTIME="`date +%s`"; echo "$f,$i,$SUCCESS,$STARTTIME,$ENDTIME,$(($ENDTIME-$STARTTIME))" >> times_orc.csv; done; done;
How do I run the queries with Presto?
presto --schema tpcds_orc -f queries/query12.sql
You can run all the queries 10 times with presto with the following command,
for f in queries/*.sql; do for i in {1..10} ; do STARTTIME="`date +%s`"; presto --schema tpcds_orc -f $f > $f.run_$i.out 2>&1 ; SUCCESS=$? ; ENDTIME="`date +%s`"; echo "$f,$i,$SUCCESS,$STARTTIME,$ENDTIME,$(($ENDTIME-$STARTTIME))" >> times_orc.csv; done; done;
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent