Reading of files with fields of type required AND optional Int64 about parquet4s HOT 5 CLOSED

from parquet4s.

Thank you.
This doesn't work if the repetition_type is required.
But I found another possibility: read the parquet as generic RowParquetRecord and then get the field as Long. Drawback is that projection is not possible. But in my use-case there is no or only little performance impact.

from parquet4s.

Hmm, I assume that we misunderstood each other and you mean that you have files with different schemas. One set of files has required fields and the second set has optional fields. And you try to read those files at once in a single job. Am I right?
In such a situation you cannot use projection at all, because by defining projection you enforce your own schema for the file. If your schema does not match the schema of the column that you read then you will get an error.
However, if you will read the data without projection, either by generic records (RowParquetRecord) or by case class (use Options) then all should be fine.
Another way is to group your files so that you have homogenous files in each group and read them separately with projection. Then use Akka Stream or experimental ETL of Parquet4s to concat the streams. Check the example: https://github.com/mjakubowski84/parquet4s/blob/master/examples/src/main/scala/com/github/mjakubowski84/parquet4s/core/ColumnProjectionAndDataConcatenationApp.scala

from parquet4s.

You are right in what I am trying to do.
ParquetReader.as[CaseClass] works even without wrapping the Long in Option and also with a case class that contains a subset of the available fields. But I assume that it is not as efficient as projection (at least theoratically)?
However that is a good enough solution. Thanks again.

from parquet4s.

But I assume that it is not as efficient as projection

Yes, it does no projection, it just decodes the whole set of data to a smaller class

from parquet4s.