compatible parquet-hadoop with spark3.1 about parquet4s HOT 3 CLOSED

Hi @conderls!
I would prefer not to add such complexity to the library.

Considering that parquet-hadoop in version 1.10.1 is five years old, I recommend you upgrade it. You may not even need to upgrade Spark. Otherwise, you still have the option of shading.

from parquet4s.

Hi @conderls! I would prefer not to add such complexity to the library.

Considering that parquet-hadoop in version 1.10.1 is five years old, I recommend you upgrade it. You may not even need to upgrade Spark. Otherwise, you still have the option of shading.

1. I agree to keep pace with the latest version, but the v1.12.x add the getPath() method breaks the compatibility and just return the path.
2. As spark-sql 3.1.2 depends on parquet-hadoop as provided scope, if without upgrade spark, then I may not able to upgrade parquet-hadoop;
3. the spark-sql depends on parquet-hadoop 1.10.1 and parquet4s depends on 1.13.1, the shading option may not work here?

from parquet4s.

2. Check out https://hadoopsters.com/how-to-override-a-spark-dependency-in-client-or-cluster-mode-2860a64ad1d5
3. In this option, you should shade parquet4s' dep

from parquet4s.