How to Enable Glue Hive MetaStore with EMR CLI about amazon-emr-cli HOT 3 CLOSED

Hi @soumilshah1995 - The option to enable the Glue catalog is a spark-submit config option. You can add it using --spark-submit-opts.

The specific option is:

--conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory

So in your case, instead of --spark-submit-opts "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer", you'll have --spark-submit-opts "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"

Thanks for trying out the CLI! Let me know if anything else comes up. :)

from amazon-emr-cli.

Couple other comments:

• Your SparkSession will need to be created with .enableHiveSupport()

spark = (
    SparkSession.builder.appName("SparkSQL")
    .enableHiveSupport()
    .getOrCreate()
)

• I created a sample sql.py script in my personal cli-examples repo: https://github.com/dacort/emr-cli-examples#spark-sql--glue-data-catalog-support

from amazon-emr-cli.

@dacort thanks a alot my issue has been resolved
i am also creating crash course for community on using EMR Serverless

from amazon-emr-cli.