The go-drill from factset

SetMaxOpenConns > zookeepers

I'm running into an issue when I set SetMaxOpenConns to something greater than the number of zookeeper nodes available.

I'm not confident whether it is a bug or intentional, though my understanding was that I could create multiple connections to single cluster.

I believe the error occurs here because eindex is out of range.

go-drill/client.go

Line 173 in eb99719

    
           return newClient, newClient.ConnectEndpoint(ctx, zook.GetEndpoint(newClient.drillBits[eindex]))

Example

package main

import (
	"database/sql"
	"fmt"
	"log"
	"strings"
	"sync"

	_ "github.com/factset/go-drill/driver"
)

type AWS struct {
	key    string
	secret string
}

var maxConnections int = 6

func ConnectDrill() (*sql.DB, error) {
	props := []string{
		"zk=localhost:2181,localhost:2182,localhost:2183",
		"cluster=drillbits1",
		"schema=s3",
	}

	db, err := sql.Open("drill", strings.Join(props, ";"))
	if err != nil {
		return nil, err
	}
	if err = db.Ping(); err != nil {
		return nil, err
	}

	db.SetMaxOpenConns(maxConnections)

	return db, nil
}

func query(con *sql.DB, wg *sync.WaitGroup) {
	defer wg.Done()
	rows, err := con.Query("SELECT * FROM sys.drillbits")

	if err != nil {
		log.Panic(err)
	}

	for rows.Next() {
		rows.Scan()
	}
	fmt.Println("queried")
}

func main() {
	var wg sync.WaitGroup

	con, _ := ConnectDrill()

	wg.Add(10)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)
	go query(con, &wg)

	wg.Wait()
}

oroutine 75 [running]:
github.com/factset/go-drill.(*Client).NewConnection(0xc0000f1500, {0xa9c7f0, 0xc000024110})
	/home/john/go/pkg/mod/github.com/factset/[email protected]/client.go:173 +0x475
github.com/JohnCoene/go-drill/driver.(*connector).Connect(0x0?, {0xa9c7f0?, 0xc000024110?})
	/home/john/Golang/src/github.com/JohnCoene/go-drill/driver/connector.go:19 +0x2f
database/sql.(*DB).conn(0xc00020cd00, {0xa9c7f0, 0xc000024110}, 0x1)
	/usr/local/go/src/database/sql/sql.go:1395 +0x782
database/sql.(*DB).query(0x0?, {0xa9c7f0, 0xc000024110}, {0x9c8f44, 0x1b}, {0x0, 0x0, 0x0}, 0x0?)
	/usr/local/go/src/database/sql/sql.go:1732 +0x5d
database/sql.(*DB).QueryContext(0x0?, {0xa9c7f0, 0xc000024110}, {0x9c8f44, 0x1b}, {0x0, 0x0, 0x0})
	/usr/local/go/src/database/sql/sql.go:1710 +0xda
database/sql.(*DB).Query(...)
	/usr/local/go/src/database/sql/sql.go:1728
main.query(0x0?, 0x0?)
	/home/john/Golang/src/github.com/JohnCoene/go-drill/cmd/main.go:42 +0x8c
created by main.main
	/home/john/Golang/src/github.com/JohnCoene/go-drill/cmd/main.go:66 +0x299
exit status 2

shell returned 1

And thanks for making this package!

Use Dremio extension for native Arrow buffers

Dremio added an extension to the Drill RPC protocol to return natively Arrow buffers instead of Drill buffers so that conversion from Drill to Arrow should be unnecessary.

Protocol extension can be found here:
https://github.com/dremio/dremio-oss/blob/3e029da88441df3af7667d13011b7905bf106f1f/protocol/src/main/protobuf/UserCoordRPC.proto#L91

factset / go-drill Goto Github PK

go-drill's People

Contributors

Stargazers

Watchers

Forkers

go-drill's Issues

SetMaxOpenConns > zookeepers

Use Dremio extension for native Arrow buffers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent