Comments (12)
Hello, I believe this is a hive limitation, so it can't be done, but you can create several connections and have a query per connection
connection1, _ := gohive.Connect("hs2.example.com", 10000, "KERBEROS", configuration)
connection2, _ := gohive.Connect("hs2.example.com", 10000, "KERBEROS", configuration)
cursor1 := connection1.Cursor()
cursor2 := connection2.Cursor()
cursor1.Exec(ctx, "insert overwrite table scm.mytable PARTITION(eventdate)")
cursor2.Exec(ctx, "select * from stg.stage_table")
from gohive.
I don't think its hive limitation because it works when tried executing the same query in python using pyhs2 package.
from gohive.
For some reason I thought you wanted to run concurrent queries on the same connection.
What you want to do should be working right now, let me look into why it's not.
from gohive.
Hello, this is actually working, but you have to remove the final ;
at the end, it's the same in pyhs2, instead of
cursor.Exec(ctx, "set hive.exec.parallel = true;")
it should be:
cursor.Exec(ctx, "set hive.exec.parallel = true")
Otherwhise hive.exec.parallel
is set to true;
instead of true
. You also have can only do one SET
statement at once.
from gohive.
Hi,
Actually, queries are executed individually without a semicolon at the end by converting into a string array using the split function. (same as pyhs2/pyhive)
so tried a couple of more permutation and combination
- Each query without a semicolon at the end with a single connection and a single cursor
result: no error but nothing happens, the query "insert overwrite .." yields nothing
2 With a semicolon at the end of each query and with a single connection and a single cursor
result: error for the last query ( insert overwrite... ) - Without a semicolon at the end of each query and with a single connectin and new cursor for each query
result: no error but nothing happens, the query "insert overwrite .." yields nothing
from gohive.
Hello, I'm not sure what's going on, I would think this test proves that setting the properties works as expected. If you could provide a POC of unexpected behavior that would be helpful.
The setting are set per session, so all of them should be executed on the same connection, then the queries should be executed on this same connection, if a new connection is created you'd need to set them all again. You could set the settings here when creating the connection so you don't have to do cursor.Exec("SET hive..... = value")
for every single setting.
from gohive.
Please find attached code snippet
test .txt
from gohive.
I've been playing around with it for a bit. Everything seems fine. I've added at the end:
cursor.Exec(context.Background(), "SET")
for cursor.HasMore(context.Background()) {
fmt.Println(cursor.RowMap(context.Background()))
}
That lists all the hive properties and they are all set as expected, is this the case for you as well? I can't see any difference with pyhs2. I guess you see the difference in the contents of scm.mytable
.
from gohive.
Yes, i could see expected hive properties are set but getting an error as attached
Though the same set of queries works fine in python
map[set:SET hive.exec.dynamic.partition.mode=nonstrict]
map[set:SET hive.exec.max.dynamic.partitions=100000]
map[set:SET hive.exec.max.dynamic.partitions.pernode=100000]
map[set:SET hive.execution.engine=mr]
map[set:SET mapred.job.name=testjob]
map[set:SET mapred.job.queue.name=default]
in python follow the same flow and it works over there
error.txt
EX
multipleQueries = jobQuery["Query"].split(";")
for query in multipleQueries:
with conn.cursor() as cur:
cur.execute(query)
from gohive.
Hello, when I run this script: main.txt, it ends without failure, however when I remove the line set hive.exec.dynamic.partition.mode = nonstrict;
from the file I get the error from the previous error.txt file, something like:
panic: Error while executing query: TStatus({StatusCode:ERROR_STATUS InfoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict:17:16 org.apache.hive.service.cli.operation.Operation:toSQLException:Operation
from gohive.
There is no issue with connection state, it is maintained ... the issue was leading or trailing space in hive set command doesn't set that particular property property
from gohive.
Thank you for debugging this and reporting back
from gohive.
Related Issues (20)
- LDAP authentication with HTTPS Does Not Work HOT 8
- Exec PENDING_STATE HOT 2
- go build error HOT 1
- some problems to access to hiveserver2 without kinit HOT 2
- Cursor.totalRows is 0
- panic: runtime error: invalid memory address or nil pointer dereference HOT 1
- How to solve "Incorrect FrameSize"? HOT 6
- could we generate cpp bindings
- What to do if the hive service is actively disconnected due to timeout? HOT 5
- Incorrect frame size HOT 6
- ConnectZookeeper panic HOT 1
- use in hive 1.1.0 fetch data uncompleted HOT 1
- hive-site.xml location HOT 1
- panic: runtime error: invalid memory address or nil pointer dereference HOT 1
- Assignment Mismatch | Thrift HOT 5
- pollUntilData func panic HOT 1
- how to specify mote than one host HOT 1
- I want to connect to the Hive database with KerberOS authentication enabled on Windows. How to install GSSAPI C library on Windows? HOT 1
- HTTP(S) plain/ldap authentication connection does not support cookies and issue with the cookie handling HOT 1
- Query never sent to server if it contains set statements HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gohive.