Support Spark 3.1 about spark-hadoopoffice-ds HOT 8 OPEN

Target version 1.5.0 and support for Spark 3.1.0. Hopefully the datasource V2 api will then be stable

from spark-hadoopoffice-ds.

target version hadoopoffice 1.4.0

from spark-hadoopoffice-ds.

@jornfranke, is this issue still considered? When do you plan relising version 1.5.0? Thanks...

from spark-hadoopoffice-ds.

Yes. Normally the current version should work on Spark 3.1, it is just a matter of testing. Release by urgency. E.g. if you find out it does not work on Spark 3.1 then I will prioritize efforts to make it happen.

from spark-hadoopoffice-ds.

@jornfranke, I tried it and it works. I only found one little issue with io.file.buffer.size property. It seems like its type has changed from String to Integer so it threw java.lang.NumberFormatException: For input string: "64K" at some point I couldn't control. Overriding the value while trying to read dataset sparkSession.read().option("io.file.buffer.size", 4096) didn't help but it helped to upgrade to hadoop version 3.3.0.

from spark-hadoopoffice-ds.

valid point - i will address this and see if it can be tested with Spark 3

from spark-hadoopoffice-ds.

Sorry, this point is not an issue in HadoopOffice as we do not set io.file.buffer.size and we do not compile the Hadoop Version into the library. I assume your spark installation must require it.

from spark-hadoopoffice-ds.

There was via a forum the wish to update the dependency of Spark to 3.x to avoid notification by JFrog Artifactory X-Ray.
I will do this after some cleanups

Note: The Jfrog warning is a false positive - we do not include Spark - we have it as provided, ie HadoopOffice uses the version provided by the runtime of the user.

from spark-hadoopoffice-ds.