S3 Access Denied in Spark History Server about spark-eks HOT 4 CLOSED

Hi @batCoder95, can you share the permissions you set for the bucket?

from spark-eks.

Hi @bbenzikry,

The IAM Role that is associated with the IRSA service account has the AWS Managed Policy "AmazonS3FullAccess" attached to it. I am attaching the screenshot of same for reference. This is the same role, service-account and bucket I am using in normal spark jobs running on same EKS and they are working perfectly fine.

Please let me know if I should share more details.

from spark-eks.

Hey @bbenzikry ,

It worked by including the SPARK_HISTORY_OPTS parameter in the helm installation command. Turns out we need to set the spark configuration spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider so that the IRSA permissions are used in the pod. Below is the final helm install command that I used to make it work:

helm install spark-history-server --set rbac.create=false,serviceAccount.create=false,serviceAccount.name=svc-spark-iam,image.repository=bbenzikry/spark-eks,image.tag=spark3-latest,environment.SPARK_HISTORY_OPTS='-Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider',pvc.enablePVC=false,nfs.enableExampleNFS=false,service.type=ClusterIP,s3.enableS3=true,s3.logDirectory=s3a://spark-bucket/events/ stable/spark-history-server --namespace dev

Closing this issue. Cheers :)

from spark-eks.

Hmm, thanks for letting me know - in the forked chart we use it's specified as follows, it may have been changed.

if [ "$enableIAM" == "false" ]; then
              export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
              -Dspark.hadoop.fs.s3a.access.key=$(cat /etc/secrets/${accessKeyName}) \
              -Dspark.hadoop.fs.s3a.secret.key=$(cat /etc/secrets/${secretKeyName})";
            else
              export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \
              -Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider";
            fi;

from spark-eks.