Could Weld be used with Ballista? about ballista HOT 2 CLOSED

I could be mistaken but after skimming the paper (extremely interesting, by the way!) it seems that Weld would need be used at a lower level than Ballista - e.g. at the DataFusion or Arrow levels (I'm not familiar enough to know which). The authors mention comparison to Spark SQL in sections 7 and 8.5:

Spark SQL natively generates Java bytecode, and uses a HyPer-like code generation process to pipeline code from different operators. In our Weld integration, we updated each Spark SQL operator to emit single Weld IR fragment encapsulating its computation, and relied on Weld’s optimizer to fuse and co-optimize these fragments.

It sounds like they're talking about replacing the JIT'd code that Spark uses, which (I think?) corresponds to the code executed in each part of the physical plan. So perhaps Ballista may be responsible for actually calling the Weld optimizer to fuse/co-optimize each fragment produced by DataFusion/Arrow?

from ballista.

@sd2k I think you may be correct. In any case, integrating with Weld is beyond the scope of this project for the immediate future at least, so I'm closing this for now.

from ballista.