Feature request: Expose partitions as a `Stream[F, Stream[F, Record]]` for FS2 about parquet4s HOT 5 CLOSED

Hi @calvinlfer,

Thank you for the contribution and a donation! Sorry for not getting back to you sooner, but you have caught me on vacation.

As you have already noticed, the code has undergone some refactoring, enabling parallel reading of individual files or partitions - as well as other improvements I started working on but notoriously have no time to finish.

Parallel reading that you proposed is on my list, and I am happy to see your PR.
However, I have a suggestion, and I would like to know your opinion. Parquet4s also contains the Akka/Pekko module, and I would like to keep the API consistency and feature parity between all modules. Therefore, I imagined adding a new optional parameter of parallelism(N) (defaulting to 1) and implementing the parallel reading of up to N files. Internally, that would be code such as yours in the case of FS2, which is a stream of streams but par-joined with an upper bound. In the case of Akka/Pekko - that would be parallel substreams.
Except for providing a consistent abstraction for multiple library modules, such an approach would prevent a situation when too many parallel connections are open, which can lead to performance degradation or depletion of resources.

from parquet4s.

Released in 2.16.0

from parquet4s.

Hey @mjakubowski84! Thank you for this amazing library and I'm glad you are taking some well earned vacation.

I definitely agree with your suggestion. we should definitely have feature parity across both Akka/Pekko and FS2.
I like the approach of parallelism(N) in the builder (which uses parJoin(N) for FS2 under the hood and the equivalent in Akka/Pekko land) as a high level API where read is an alias for readPar(1).

I would also advocate for exposing the Stream of Streams abstraction in FS2/equivalent in Akka for more more advanced users that want a handle to each file as an inner stream since parJoin variants will merge these thereby losing fine grained control.

from parquet4s.

@calvinlfer I've forgotten to ask... I mentioned you as a sponsor in Readme and the project page using your Github username and linking your Github profile. Would you prefer a different name and a link to a different page?

from parquet4s.

Thank you for adding me! What you have is perfect 😄

from parquet4s.