Overwrite mode not overwriting partitions in S3 with `viaParquet`? about parquet4s HOT 5 CLOSED

You are right! It turns out that overwrite mode has never been implemented in viaParquet. And to implement it properly, validateWritePath has to be modified, too.

from parquet4s.

Hi @jhenahan,

I am not sure what's going on. Parquet4s, by design, deletes the whole directory indicated in .write(Path(prefix)) when override mode is in use.
I recommend turning on debug logs. Maybe there's an issue with permissions?

from parquet4s.

Digging through logs, it looks like viaParquet never calls io.validateWritePath, so the rotating writer never deletes filesystems. writeSingleFile does, but can't write partitioned data. I think it might be sufficient to call validateWritePath in rotatingWriter's write definition, but I'm not sure. I'll try it out locally, but if you know better I'm very happy to take some pointers and write a PR.

from parquet4s.

diff --git i/fs2/src/main/scala/com/github/mjakubowski84/parquet4s/parquet/rotatingWriter.scala w/fs2/src/main/scala/com/github/mjakubowski84/parquet4s/parquet/rotatingWriter.scala
index a60f59a..0baadcb 100644
--- i/fs2/src/main/scala/com/github/mjakubowski84/parquet4s/parquet/rotatingWriter.scala
+++ w/fs2/src/main/scala/com/github/mjakubowski84/parquet4s/parquet/rotatingWriter.scala
@@ -517,6 +517,7 @@ object rotatingWriter {
         logger     <- Stream.eval(logger[F](this.getClass))
         eventQueue <- Stream.eval(Queue.unbounded[F, WriterEvent[F, T, W]])
         writersRef <- Stream.eval(Ref.of(Map.empty[Path, RecordWriter[F]]))
+        _          <- Stream.eval(io.validateWritePath[F](basePath, options, logger))
         rotatingWriter <- Stream.emit(
           new RotatingWriter[T, W, F](
             basePath            = basePath,

This is my local diff. Initial testing looks promising. :)

from parquet4s.

The fix released is released in v2.9.0

from parquet4s.