ackuq / spark-pit Goto Github PK

First, thank you for building this and open sourcing it. My current estimate is that the early stop sort merge provides a 25X speed up on one of our workloads compared to the union plus window function approach we were doing before. This makes make extremely happy!

This is kind of a strange github issue because I just want to ask some questions. I'm interested in potentially contributing to this and I don't want to just start bombarding you with stuff.

1. If I make PRs would they get reviewed and merged? Likely PRs:
   1. Upgrade to Spark 3.5.0
   2. Switch to using SparkSession instead of SQLContext which is deprecated.
   3. Some little edgecase bug fixes.
2. I'm confused why there is both doExecute and doProduce, since both seem to contain very similar logic. It seems like doExecute only gets used when joining more than 2 datframes or if the input dataframes have >= 98 columns (this part confuses me even more). Probably I'm just missing some context but I think its quite likely that all the little bugs I found would not have happened if there was a single implementation. I would be very interested to know why its like this.