Code Monkey home page Code Monkey logo

Comments (1)

anks2024 avatar anks2024 commented on August 28, 2024

Hey I am also trying to use the brickhouse on Spark but I am not able to to do so.
I did the following to register the function:
spark.sql("create temporary function collect as 'brickhouse.udf.collect.CollectUDAF'")

Then to use the UDAF on TempView (tab1),

tab1 schema : (issue_id : string, field: string, from_val: string)
eg:
(Issue1, priority, P1)
(Issue1, severity, Critical)
(Issue2, priority, P3)
(Issue2, severity, Medium)

spark.sql("select issue_id, g_m['priority'] as priority, g_m['severity'] as severity from ( select issue_id, collect(field, from_val) as g_m from tab1 group_by issue_id) aa")

To achieve the following:
issue_id, priority, severity
Issue1, P1, Critical
Issue2, P3, Medium

But what I get is:
Traceback (most recent call last):
File "", line 1, in
File "/opt/..../pyspark/sql/session.py", line 545, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/opt/..../java_gateway.py", line 1133, in call
File "/opt/..../utils.py", line 73, in deco
raise ParseException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.ParseException: u"\nmismatched input 'from' expecting {, 'WHERE', 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT', 'SORT', 'CLUSTER', 'DISTRIBUTE'}(line 1, pos 88)\n\n== SQL ==\nselect issue_id, created_date, g_m['priority'] as priority, g_m['severity'] as severity from ( select issue_id, to_date(i_created_date) as created_date, collect(field, from_val) as g_m from tab1 group_by issue_id, created_date) aa\n----------------------------------------------------------------------------------------^^^\n"

Any idea how I can solve this error?

from brickhouse.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.