Databricks - GCP Key Missing in Multi-Node Cluster about spark-bigquery HOT 3 CLOSED

not sure how to accomplish this in a notebook, but I've found I have to include my json key with the --files /path/to/key.json when running spark-submit. that should get it on all the worker nodes where it is readable.

from spark-bigquery.

Hi guys

@kurtmaile you are absolutely right, the worker node cannot find the gcp file.

What you can do is create an init script which will load the credentials file when a cluster is created

Here is an example script

dbutils.fs.rm("dbfs:/databricks/init/copy-google-key-json.sh",true)
dbutils.fs.put("/databricks/init/copy-google-key-json.sh","""
#!/bin/bash 
cat <<EOF > /databricks/key.json
{paste json here}
EOF
""")

I hope this helps!

from spark-bigquery.

Hi guys,

Awesome thanks for this info this helps greatly!
I will give the init feature a go today, much appreciated once again for the speedy response.

Cheers

from spark-bigquery.