Code Monkey home page Code Monkey logo

go-impala's People

Contributors

alileza avatar ikkeps avatar intamyuto avatar koblas avatar liracle avatar noahhl avatar shaloi avatar timarmstrong avatar vishjosh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

go-impala's Issues

compilation with thrift v0.14.1 failed

github.com/bippio/go-impala/services/cli_service

vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:721:37: not enough arguments in call to iprot.ReadStructBegin
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:727:57: not enough arguments in call to iprot.ReadFieldBegin
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:739:29: not enough arguments in call to iprot.Skip
have (thrift.TType)
want (context.Context, thrift.TType)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:758:33: not enough arguments in call to iprot.ReadFieldEnd
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:762:32: not enough arguments in call to iprot.ReadStructEnd
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:769:29: not enough arguments in call to iprot.ReadI32
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:778:32: not enough arguments in call to iprot.ReadString
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:790:35: not enough arguments in call to oprot.WriteStructBegin
have (string)
want (context.Context, string)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:796:33: not enough arguments in call to oprot.WriteFieldStop
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:798:33: not enough arguments in call to oprot.WriteStructEnd
have ()
want (context.Context)
vendor/github.com/bippio/go-impala/services/cli_service/cli_service.go:798:33: too many errors

Compilation finished with exit code 2

Query Timeout having no effect

`
package main

import (
"context"
"database/sql"
"log"

imp "github.com/bippio/go-impala"

)

func main() {

opts := imp.DefaultOptions

opts.Host = "some_host"
opts.Port = "21050"

// enable LDAP authentication:
opts.UseLDAP = true
opts.Username = "xx"
opts.Password = "yy"

// enable TLS
opts.UseTLS = true
opts.CACertPath = "/opt/bipp/certs/zz.pem"

opts.QueryTimeout = 1

connector := imp.NewConnector(&opts)
db := sql.OpenDB(connector)
defer db.Close()

ctx := context.Background()

rows, err := db.QueryContext(ctx, "Loooong running query (use a self-join to simulate one)")
if err != nil {
	log.Fatal(err)
}
defer rows.Close()

type Record struct {
	rec1   string
	//rec2  string
	//rec3 int
}

var record Record

results := make([]Record, 0)
for rows.Next() {
	if err := rows.Scan(&record.rec1); err != nil {
		log.Fatal(err)
	}
	results = append(results, record)
}
if err := rows.Err(); err != nil {
	log.Fatal(err)
}
log.Println("Results", results)

}

`

Tried the above code on a long running query, the query timeout never kicked in. Are we doing something wrong or is there some other issue ?

Suspected file handle leak , server using the driver becomes unresponsive

We are using the driver and there are scenarios where we make a lot of queries to a impala backend.

When we looked at the open file handles when the server becomes unresponsive - we see that there are a lot of open file handles.

We are wondering if this has something to do with the driver not managing its connections correctly.

Errors from cli_service.go

Having trouble getting and building (cli_service.go). This maybe similar to Thrift version Issue #24. I tried to resolve with solution in #24 but was unsuccessful. I even tried 'go clean -cache' to be sure but no luck.

mark@MarkHughes:~/go/src/DBL_168/nusiem-device-create-hive-tables$ go build -v -x
WORK=/tmp/go-build437803886
github.com/bippio/go-impala/services/cli_service
mkdir -p $WORK/b122/
cat >$WORK/b122/importcfg << 'EOF' # internal

import config

packagefile bytes=/usr/local/go/pkg/linux_amd64/bytes.a
packagefile context=/usr/local/go/pkg/linux_amd64/context.a
packagefile database/sql/driver=/usr/local/go/pkg/linux_amd64/database/sql/driver.a
packagefile errors=/usr/local/go/pkg/linux_amd64/errors.a
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile github.com/apache/thrift/lib/go/thrift=/home/mark/.cache/go-build/66/66bfbb0508076af1b7ccdac2ef12b2a7adffd3de8fcb1355c3ca10528079b032-d
packagefile reflect=/usr/local/go/pkg/linux_amd64/reflect.a
EOF
cd /home/mark/go/src/DBL_168/nusiem-device-create-hive-tables
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b122/pkg.a -trimpath "$WORK/b122=>" -p github.com/bippio/go-impala/services/cli_service -complete -buildid akEEO3jNMNkzO7vPVkKV/akEEO3jNMNkzO7vPVkKV -goversion go1.16 -D "" -importcfg $WORK/b122/importcfg -pack -c=4 /home/mark/go/pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/GoUnusedProtection__.go /home/mark/go/pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service-consts.go /home/mark/go/pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go

github.com/bippio/go-impala/services/cli_service

../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:721:37: not enough arguments in call to iprot.ReadStructBegin
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:727:57: not enough arguments in call to iprot.ReadFieldBegin
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:739:29: not enough arguments in call to iprot.Skip
have (thrift.TType)
want (context.Context, thrift.TType)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:758:33: not enough arguments in call to iprot.ReadFieldEnd
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:762:32: not enough arguments in call to iprot.ReadStructEnd
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:769:29: not enough arguments in call to iprot.ReadI32
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:778:32: not enough arguments in call to iprot.ReadString
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:790:35: not enough arguments in call to oprot.WriteStructBegin
have (string)
want (context.Context, string)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:796:33: not enough arguments in call to oprot.WriteFieldStop
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:798:33: not enough arguments in call to oprot.WriteStructEnd
have ()
want (context.Context)
../../../pkg/mod/github.com/bippio/[email protected]+incompatible/services/cli_service/cli_service.go:798:33: too many errors
mark@MarkHughes:~/go/src/DBL_168/nusiem-device-create-hive-tables$

Design Change - Move to HS2 interface to Impala rather than beeswax

Suggestion from @timarmstrong .
@intamyuto - let us discuss if this is something we can start working on in parallel . We can get a lot of help from the community and Tim has already pointed us to a good set of resources . His C++ client seems to be very relevant to our go driver.

Suggestion Details

I'd strongly suggest to use the HS2 interface to Impala (port 21050 by default) rather than beeswax. Beeswax is a legacy thing that's really only kept alive because impala-shell hasn't been ported to use HS2. I've been starting to advocate for doing that work and removing beeswax from Impala in the future. Generally HS2 makes a lot more sense, is exercised by more different clients, and should be easier to work with. E.g. the text-based beeswax protocol can't handle tabs properly - https://github.com/bippio/go-impala/blob/master/rowset.go#L171 and you don't need to do string parsing in the client - https://github.com/bippio/go-impala/blob/master/rowset.go#L203. I think the hard stuff like SSL/LDAP should all be the same.

Here's the thrift interface - https://github.com/apache/impala/blob/master/common/thrift/hive-1-api/TCLIService.thrift#L1138. You get back the data as a set of columns in the native types - https://github.com/apache/impala/blob/master/common/thrift/hive-1-api/TCLIService.thrift#L395. Impyla and the C++ hs2client (not finished, but quite clean code) both use that interface - https://github.com/cloudera/impyla, https://github.com/cloudera/hs2client, along with the standard JDBC and ODBC clients.

Reestablish connection after EPIPE (Broken pipe)

Hi!

First, thanks for the great work :)

I came across the issue that when a "broken pipe" error thrown by the net package, the driver keeps using the connection instead of either abandon it or re-establish a new connection by closing the existing one and opening a new one. The problem will persist forever unless connection max lifetime is set in that case the bad connection will be closed at time T when it reaches its lifetime.

Here is how you can re-produce the scenario manually.

  1. Run example program
  2. Take the port number from local address. For example 1234 based on the following example
sudo netstat -anp | grep 21050
tcp        0      0 x.x.x.x:1234     y.y.y.y:21050    ESTABLISHED 100000/impala-example
  1. Kill the connection to create a broken pipe senario as follows
sudo tcpkill -i eth0 -9 port 1234
  1. Make your example program execute a query and you will get "broken pipe" errors

I tried to see if closing and opening a connection in "broken pipe" state fixes the problem during OpenSession and indeed it will fix the issue but I'm not sure if that is the right place to put the fix.

supper kerberos

I rewrite TSaslTransport for kerberos of GSSAPI by github.com/jcmturner/gokrb5 for my project
but I not sure Should I to comit ,because is different OPTION

Error in cli_service.go: argument 1 has incompatible type (missing method 'Deadline')

Hi.

Here is what I got:

$ go build -i -v -x
WORK=/tmp/go-build219060932
github.com/bippio/go-impala/sasl
mkdir -p $WORK/b035/
cd $WORK
/home/oceanfish81/gollvm_dist/bin/llvm-goc -fgo-importcfg=/dev/null -c -x c - -o /dev/null || true
github.com/bippio/go-impala/services/cli_service
mkdir -p $WORK/b035/importcfgroot/github.com/apache/thrift/lib/go
mkdir -p $WORK/b033/
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/apache/thrift/lib/go/libthrift.a $WORK/b035/importcfgroot/github.com/apache/thrift/lib/go/libthrift.a
cd /home/oceanfish81/go/src/github.com/bippio/go-impala/sasl
/home/oceanfish81/gollvm_dist/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -fgo-pkgpath=github.com/bippio/go-impala/sasl -o $WORK/b035/go.o -I $WORK/b035/importcfgroot ./client.go ./plain.go ./sasl.go ./transport.go
mkdir -p $WORK/b033/importcfgroot/github.com/apache/thrift/lib/go
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/apache/thrift/lib/go/libthrift.a $WORK/b033/importcfgroot/github.com/apache/thrift/lib/go/libthrift.a
cd /home/oceanfish81/go/src/github.com/bippio/go-impala/services/cli_service
/home/oceanfish81/gollvm_dist/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -fgo-pkgpath=github.com/bippio/go-impala/services/cli_service -o $WORK/b033/go.o -I $WORK/b033/importcfgroot ./GoUnusedProtection__.go ./cli_service-consts.go ./cli_service.go
echo ' .section .go.buildid,"e"' >> $WORK/b035/_buildid.s
echo ' .byte 0x65,0x5a,0x48,0x72,0x6e,0x6f,0x37,0x55' >> $WORK/b035/_buildid.s
echo ' .byte 0x38,0x6d,0x50,0x79,0x2d,0x5a,0x55,0x67' >> $WORK/b035/_buildid.s
echo ' .byte 0x62,0x74,0x65,0x35,0x2f,0x65,0x5a,0x48' >> $WORK/b035/_buildid.s
echo ' .byte 0x72,0x6e,0x6f,0x37,0x55,0x38,0x6d,0x50' >> $WORK/b035/_buildid.s
echo ' .byte 0x79,0x2d,0x5a,0x55,0x67,0x62,0x74,0x65' >> $WORK/b035/_buildid.s
echo ' .byte 0x35' >> $WORK/b035/_buildid.s
echo ' .section .note.GNU-stack,"",@progbits' >> $WORK/b035/_buildid.s
echo ' .section .note.GNU-split-stack,"",@progbits' >> $WORK/b035/_buildid.s
echo '' >> $WORK/b035/_buildid.s
cd /home/oceanfish81/go/src/github.com/bippio/go-impala/sasl
/home/oceanfish81/gollvm_dist/bin/llvm-goc -xassembler-with-cpp -I $WORK/b035/ -c -o $WORK/b035/_buildid.o -D GOOS_linux -D GOARCH_amd64 -D GOPKGPATH=github.x2ecom..z2fbippio..z2fgo..z2dimpala..z2fsasl -m64 $WORK/b035/_buildid.s
ar rcD $WORK/b035/pkg.a $WORK/b035/go.o $WORK/b035/_buildid.o
/home/oceanfish81/gollvm_dist/tools/buildid -w $WORK/b035/pkg.a # internal
cp $WORK/b035/pkg.a /home/oceanfish81/.cache/go-build/6c/6c13564ac1673b97012d5556cd1ad55e6d447aed60c32985670182c34e830221-d # internal
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/bippio/go-impala/
mv $WORK/b035/pkg.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/bippio/go-impala/libsasl.a
rm -r $WORK/b035/

github.com/bippio/go-impala/services/cli_service

../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:721:21: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:727:42: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:739:30: error: argument 1 has incompatible type (missing method 'Deadline')
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:739:24: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:749:30: error: argument 1 has incompatible type (missing method 'Deadline')
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:749:24: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:754:28: error: argument 1 has incompatible type (missing method 'Deadline')
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:754:22: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:758:20: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:762:18: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:769:21: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:778:21: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:790:55: error: argument 1 has incompatible type (type has no methods)
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:790:18: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:796:18: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:798:18: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:805:45: error: argument 1 has incompatible type (type has no methods)
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:805:56: error: argument 2 has incompatible type (cannot use type int as type string)
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:805:20: error: not enough arguments
../../go/src/github.com/bippio/go-impala/services/cli_service/cli_service.go:807:30: error: argument 1 has incompatible type (type has no methods)

go-impala doesn't close queries

Well-behaved clients are expected to close queries via the close() method when they are done with them: https://github.com/apache/impala/blob/master/common/thrift/beeswax.thrift#L179. This releases all resources associated with the queries and generally avoids open queries from accumulating on the server.

I experimented with impala-go and it leaves all of the queries dangling until the client disconnects.
I0211 08:43:31.541136 30975 impala-server.cc:2012] Connection from client ::ffff:127.0.0.1:38950 closed, closing 1 associated session(s) I0211 08:43:31.541154 30975 impala-server.cc:1260] Closing session: d94d857aa1bb0ada:8732f4f69745269b I0211 08:43:31.541177 30975 impala-server.cc:1142] UnregisterQuery(): query_id=f84b83f4b6db701a:7457100f00000000 I0211 08:43:31.541182 30975 impala-server.cc:1249] Cancel(): query_id=f84b83f4b6db701a:7457100f00000000 I0211 08:43:31.541781 30975 impala-server.cc:1142] UnregisterQuery(): query_id=87424b90dbea4d1f:7796c34b00000000 I0211 08:43:31.541787 30975 impala-server.cc:1249] Cancel(): query_id=87424b90dbea4d1f:7796c34b00000000 I0211 08:43:31.542315 30975 impala-server.cc:1142] UnregisterQuery(): query_id=d24932ba436cdf1c:78fc87f000000000 I0211 08:43:31.542320 30975 impala-server.cc:1249] Cancel(): query_id=d24932ba436cdf1c:78fc87f000000000

I don't see any calls to close() in the source code of impala-go which explains that.

Unfortunately the contract around query unregistration is a little messy right now - in some cases, if the query encounters an error, it will return the error to the client on the next RPC and then automatically close the query. This only happens in some cases. I'd recommend calling close() and ignoring any QueryNotFoundException returned by close()

suspect Ping() does not "establishing a connection if necessary"

"Ping verifies a connection to the database is still alive, establishing a connection if necessary."
But I've got an error message from it - "write tcp 172.22.0.2:38686->10.2.44.115:21050: write: broken pipe"

Does Ping() works wrong or its description is incorrect?

execute query while no response

I tried this demo on CDH cluster with no authorization, code like this
func main() {
host := "cloudera.demo"
port := 21050

opts := impalathing.DefaultOptions


con, err := impalathing.Connect(host, port, &opts)
if err != nil {
	log.Fatal(err)
}

query, err := con.Query(context.Background(), "select 1")

startTime := time.Now()
results := query.FetchAll(context.Background())
log.Printf("Fetch %d rows(s) in %.2fs", len(results), time.Duration(time.Since(startTime)).Seconds())

}

while the query is hunging in the step "con.Query"๏ผŒno response

How to use( use databases)

My function needs to select the database to execute sql, otherwise an error will be reported:

error:QueryContest:AnalysisException: Could not resolve table reference: 'user_tag_view_user_group_hyyhhd4dbbzyws_production'

However, it does not take effect after use is executed

if _, err := DB.ExecContext(ctx, "use rawdata"); err != nil {
         return res, fmt.Errorf("use rawdata:%v", err)
}

Couldn't exec impala function at INSERT Statement

Just like title, When I exec insert statement like
insert into table TABLE_A (id,user_name) select id,user_name from TABLE_B

It good to work, But if I give a function like
insert into table TABLE_A (id,user_name,age) select id,user_name,cast(age as string) from TABLE_B
No fail message and no data insert

panic in GetResultSetMetadata when schema object is nil

Panic has been encountered in some cases with the following stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdf45ce]

goroutine 1769811 [running]:
bippv2/server/vendor/github.com/bippio/go-impala/hive.(*Operation).GetResultSetMetadata(0xc003910ce0, 0x197ebe0, 0xc0023e4ae0, 0xc000a2a640, 0x92, 0xc003910ce0)
        /home/knight/go/src/bippv2/server/vendor/github.com/bippio/go-impala/hive/operation.go:42 +0x1ae
bippv2/server/vendor/github.com/bippio/go-impala.query(0x197ebe0, 0xc0023e4ae0, 0xc0041c8990, 0xc000a2a640, 0x92, 0xc000a2a640, 0x92, 0x0, 0xc0014fd770)
        /home/knight/go/src/bippv2/server/vendor/github.com/bippio/go-impala/statement.go:117 +0x9f
bippv2/server/vendor/github.com/bippio/go-impala.(*Conn).QueryContext(0xc0014fd770, 0x197ebe0, 0xc0023e4ae0, 0xc000a2a640, 0x92, 0x22522d0, 0x0, 0x0, 0x203000, 0x15e0c00, ...)
        /home/knight/go/src/bippv2/server/vendor/github.com/bippio/go-impala/connection.go:69 +0x113
database/sql.ctxDriverQuery(0x197ebe0, 0xc0023e4ae0, 0x7fc2fc13e8e0, 0xc0014fd770, 0x0, 0x0, 0xc000a2a640, 0x92, 0x22522d0, 0x0, ...)
        /usr/local/go/src/database/sql/ctxutil.go:48 +0x227
database/sql.(*DB).queryDC.func1()
        /usr/local/go/src/database/sql/sql.go:1643 +0x1de
database/sql.withLock(0x196da40, 0xc004fb4750, 0xc0018adb60)
        /usr/local/go/src/database/sql/sql.go:3284 +0x69
database/sql.(*DB).queryDC(0xc00108c5b0, 0x197ebe0, 0xc0023e4ae0, 0x0, 0x0, 0xc004fb4750, 0xc003910cb0, 0xc000a2a640, 0x92, 0x0, ...)
        /usr/local/go/src/database/sql/sql.go:1638 +0x5f4
database/sql.(*DB).query(0xc00108c5b0, 0x197ebe0, 0xc0023e4ae0, 0xc000a2a640, 0x92, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
        /usr/local/go/src/database/sql/sql.go:1621 +0x136
database/sql.(*DB).QueryContext(0xc00108c5b0, 0x197ebe0, 0xc0023e4ae0, 0xc000a2a640, 0x92, 0x0, 0x0, 0x0, 0x5640, 0x0, ...)
        /usr/local/go/src/database/sql/sql.go:1598 +0xd1

cannot set mem_limit query option

I am trying to connect to a cloudera instance of impala, which expects me to set mem_limit as a query option, otherwise it will refuse too large running of queries.

I was attempting to implement this last week with the beeswax version and had forked to be able to set beeswax configuration options, but I was having problems getting it to pass the right string.

With the switch to HS2, I thought by swapping to the ODBC, I could send it as a separate query before and it would keep it in session potentially for the user, but this is not the case.

I can see where the mem_limit option is part of the thrift definition

TImpalaQueryOptions_MEM_LIMIT TImpalaQueryOptions = 4
, but I cannot seem to sort out how to properly send it, and could not sort out how it used those or the default options in the code.

Can someone clarify how to properly make those settings?

got not all rows in response. cursor is needed?

I make simple request to my database:
rows, err := db.QueryContext(ctx, "select userid from pl_session where year = 2022 and month=01 and day = 27")
Result should be 700k rows (It is verified via db web interface).
But via go-impala I have only 160k ones.
Why?
Is it some-how related to requests with cursor?

Issues when query returns lots of records/rows

impala.go.txt

We are running the attached client program to iterate all the DBs and then iterate the tables for each DB.

We realized there are 2 issues :

  • Issue 1: Some DBs had more than 1K tables .

    • Interestingly the Fetch call below in rowset's Next() function would succeed (the response's data would be arbitrarily truncated which goes unnoticed) , but the RowSet's Next() call would finally fail with following error (which also goes unnoticed by the client - all that the client knows is that there are no results to iterate through)
      GetResultsMetadata failed: get_results_metadata: wrong method name
    • resp, err := r.client.Fetch(ctx, r.handle, false, 1000000)
    • The solution seems to make sure that the fetch call requests lower number of rows , however it is not clear what the right number should be . 100 seems to work . 1000 seems to be too high.
  • Issue 2 , Interestingly after the Next() call errors out, it just returns false and the client code moves on to the next query . However all subsequent queries on that connection fail with the following error . This is extremely problematic, ideally we should be able to reset the state and start a new query even if earlier queries error out.

    • error in querying database X: query: wrong method name

@timarmstrong - I would need some suggestion from you on both the issues .

Both issues seem to be stemming due to the driver's incorrect understanding of how to use the thrift interface/services.
Any help will be appreciated to point us to a more robust way of doing this without changing the current dependency on beeswax.

@intamyuto FYI

Multiple identical parameters in SQL Query

Hi there,

First off thanks for your work, library looks good ๐Ÿ‘
However I have one question. Suppose we have following sql-statement:
SELECT name FROM persons WHERE name = ? or lastname = ?

I want to pass the same parameter for both ?. Is this somehow possible to pass the parameter once and have it for each "?" replaced?

My problem is that I dont know how much "?" are in my file and I want to pass for each the same value.

Do you have something which I missed? Thanks a lot

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.