Comments (1)
Hey I am also trying to use the brickhouse on Spark but I am not able to to do so.
I did the following to register the function:
spark.sql("create temporary function collect as 'brickhouse.udf.collect.CollectUDAF'")
Then to use the UDAF on TempView (tab1),
tab1 schema : (issue_id : string, field: string, from_val: string)
eg:
(Issue1, priority, P1)
(Issue1, severity, Critical)
(Issue2, priority, P3)
(Issue2, severity, Medium)
spark.sql("select issue_id, g_m['priority'] as priority, g_m['severity'] as severity from ( select issue_id, collect(field, from_val) as g_m from tab1 group_by issue_id) aa")
To achieve the following:
issue_id, priority, severity
Issue1, P1, Critical
Issue2, P3, Medium
But what I get is:
Traceback (most recent call last):
File "", line 1, in
File "/opt/..../pyspark/sql/session.py", line 545, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/opt/..../java_gateway.py", line 1133, in call
File "/opt/..../utils.py", line 73, in deco
raise ParseException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.ParseException: u"\nmismatched input 'from' expecting {, 'WHERE', 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT', 'SORT', 'CLUSTER', 'DISTRIBUTE'}(line 1, pos 88)\n\n== SQL ==\nselect issue_id, created_date, g_m['priority'] as priority, g_m['severity'] as severity from ( select issue_id, to_date(i_created_date) as created_date, collect(field, from_val) as g_m from tab1 group_by issue_id, created_date) aa\n----------------------------------------------------------------------------------------^^^\n"
Any idea how I can solve this error?
from brickhouse.
Related Issues (20)
- add jar: jar does not exist HOT 2
- Not able to mvn package HOT 2
- Logger level settings HOT 1
- mvn package error HOT 1
- collect UDAF just get the last value
- Software grant to the ASF HOT 17
- Links in Downloads page are broken
- HyperLogLog Tests Failing with Casting Issue HOT 1
- Latest Version of Apache Hive (2.3.2) has removed org.apache.hadoop.hive.serde2.SerDe causing ClassNotFoundException.
- All vector UDFs that take in lists will always throw an IndexOutofBoundsException
- VectorMagnitudeUDF takes in a scalar but is never used
- union_vector_sum throws java.lang.IndexOutOfBoundsException
- Add code examples to "Cook Book" HOT 1
- Mark JsonFactory in ToJsonUDF as transient HOT 1
- Hive already have the UDAF:collect_list(), why implement collect()? HOT 3
- How do I install brickhouse in my EMR cluster? HOT 1
- Brickhouse to_json UDF does not support Hive's Decimal type HOT 1
- CollectUDAF makes the tasks fail because of out of memory
- what is the second arg of cast_arrary
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from brickhouse.