hbutani / spark-sql-macros Goto Github PK

Great stuff, and thanks for sharing! This could greatly improve Spark UDF performance.

I just started to give a try, but run into the following warning. This warning, of course, failed the translation, thus there is no filter pushdown "PushedFilters: []":
scala> spark.registerMacro("intUDM", spark.udm((i: Int) => { | val j = 2 | i + j | })) <console>:27: warning: Not able to translate Value definition: 'val i: scala.Int = _}' spark.registerMacro("intUDM", spark.udm((i: Int) => { ^ scala>
I am using Spark 3.1.1 and Scala 2.12.10, not sure if these versions are okay.

Your help is highly appreciated!

-Yong

I went through the docs and some of the source, and that was a fascinating edifying read; obviously having first-class UDFs would open up all kinds of new development opportunities with Spark, so this is potentially very exciting(!)

I'd love to give it a go for am ancillary job to see for myself, but don't quite have the time to really delve into the gory details of a brand new library. I didn't see much setup instructions in the wiki, and I'd be glad to be corrected, I was just wondering if I should hold off as you're still adding some documentation or if it should be pretty straightforward to work with in its current state.

Would love to loop back to this and help with said documentation/debugging, just not right at this moment. Cool stuff nonetheless, and I'll be sure to follow it.