qustavo / sqlhooks Goto Github PK
View Code? Open in Web Editor NEWAttach hooks to any database/sql driver
License: MIT License
Attach hooks to any database/sql driver
License: MIT License
If we don't implement QueryerContext or Queryer interface for connection, db.Query() will always do statement "prepare" and statement "exec". These extra roundtrips can add unneeded latency.
Run this test program and check in wireshark: https://play.golang.org/p/wvmsGliLPpz
A change like this fixes this: surki@98e9bd8
I could send a PR if you would like this fixed here
It'd be easier to integrate with other wrappers if a WrapDriver function was available, similar to ExpansiveWorlds/instrumentedsql
The latest version has implemented some new interfaces defined in Go 1.8. It would be useful if the Go version required is mentioned in the README.
In order to combine multiple hooks (log+tracing) a new (compose) hook can be implemented. It should accept a list of Hooks and chain them together.
Go 1.8 has added a bunch of new interfaces that need to be wrapped, all the context related functions pretty much.
Tracing hooks seems to be the perfect use case for sqlhooks
When connecting over a high-latency network, it becomes visible in my tracing that Prepare
takes a while to execute when using github.com/go-sql-driver/mysql
.
Does it sound like it makes sense to add hooks around Prepare
, too?
V0 has some benchmarks, it would be good to have them back on v1
ATM no linters are set, anything will do for now. Code could be improved later
Great work! I think this could be great for data change notification events to various parts of an application. Is it possible to make the hooks pre or post execution with the ability to cancel a pre-hook query? For example, a use case might be to stop any query that contains "CREATE TABLE" for users that don't have that permission.
I've recently tried out sqlhooks and encountered a problem:
(/Users/dobegor/go/src/smartback/v2/service/label_service.go:65)
[2018-05-23 14:51:15] Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"labels" WHERE (user_id = 0 AND is_system = 1 AND label IN (?,?,?,?))' at line 1
(/Users/dobegor/go/src/smartback/v2/service/label_service.go:65)
[2018-05-23 14:51:15] [2.00ms] SELECT * FROM "labels" WHERE (user_id = 0 AND is_system = 1 AND label IN ('Inbox','Archive','Trash','Snoozed'))
As you can see, for some reason requests went into mysql without substitution.
This only occurs if I wrap the driver with sqlhooks:
sql.Register("mysqlWithHooks", sqlhooks.Wrap(&mysql.MySQLDriver{}, &Hooks{}))
db, err := gorm.Open("mysqlWithHooks", config.DB.DSN())
if err != nil {
return stacktrace.Propagate(err, "Failed to connect to DB")
}
I wonder if I'm doing something wrong here.
I'm using the hooks to create audit_logs for my project, I'm using the idea of this lib https://github.com/gmhafiz/audit but in this lib, the author uses a personalized version of sqlhooks (u can check this in the go.mod) that receive a driver.Result that has the LastInsertId() that he uses to populate the RowId in his audit struct.
My goal is to be able to use these Hooks to get the ID of the last record inserted on my Postgres database in a similar way.
i add onError
hook, add collect driver.ErrSkip
error, how about add error check before call onError
There have been a few changes to sqlhooks. We should probably tag the latest commit. I noticed this while trying out the dep tool.
I can't start transaction with non default isolation level because connection returned from sqlhooks.Conn wrapper doesn't implement driver.ConnBeginTx
interface https://github.com/golang/go/blob/master/src/database/sql/ctxutil.go#L100
I would like to execute SQL commands inside the hook methods. These commands must be on the same transaction of the original command. I mean the one triggered by the programmer using the database/sql API. I have to do this transparently. Do you think it is feseable and straightforward to implement? I guess I would have to add extra parameters to the hook methods' signatures. However, I am wondering how I could append these extra comands to the ongoing transaction.
The hooks get entered twice for each sql query because from my understanding, the first hit to the hook get cancelled because of driver.ErrSkip "driver: skip fast-path; continue as if unimplemented"
It seems to happen with sqlx
.
More details is in the discussion over at data-dog DataDog/dd-trace-go#270
Right now a quick fix would be checking for driver.ErrSkip
and ignore it in every execs
and query
results, err := conn.execContext(ctx, query, args)
- if err != nil {
+ if err != nil && err != driver.ErrSkip {
return results, handlerErr(ctx, conn.hooks, err, query, list...)
}
The other fix (from the thread) is to set interpolateParams=true
is the database dsn.
Benchmarks show a 2x and 3x overhead (mysql and psql). This should decrease.
I was hoping the after hook would include the resulting query, as it is actually passed to the database.
So instead of showing the original query like
'select id, first, last from users where id=?', '12'
It would include
'select id, first, last from users where id=12'
Useful for debugging edge cases, logging of all actual statements (to rebuild a database or specific situation) and other things.
I've got a panic when using your library (2.1.0) with github.com/microsoft/[email protected]
sql driver.
Stack trace
/usr/local/go/src/runtime/panic.go:770 +0x124
github.com/qustavo/sqlhooks/v2.composed.OnError({0x14000a5a650?, 0x2?, 0x104ad1740?}, {0x1051a44d8, 0x140009884e0}, {0x10517a120, 0x14000acade0}, {0x1043e2b1a, 0x23}, {0x1400073c940, ...})
/Users/vvinogradov/go/pkg/mod/github.com/qustavo/sqlhooks/[email protected]/compose.go:52 +0xfc
github.com/qustavo/sqlhooks/v2.handlerErr({0x1051a44d8?, 0x140009884e0?}, {0x1051851f0?, 0x14000915908?}, {0x10517a120, 0x14000acade0}, {0x1043e2b1a?, 0x23?}, {0x1400073c940?, 0x2?, ...})
/Users/vvinogradov/go/pkg/mod/github.com/qustavo/sqlhooks/[email protected]/sqlhooks.go:32 +0x8c
github.com/qustavo/sqlhooks/v2.(*Stmt).QueryContext(0x140009884b0, {0x1051a4430, 0x106abf780}, {0x14000534a50, 0x2, 0x2})
/Users/vvinogradov/go/pkg/mod/github.com/qustavo/sqlhooks/[email protected]/sqlhooks.go:312 +0x1c4
database/sql.ctxDriverStmtQuery({0x1051a4430, 0x106abf780}, {0x1051a56c8, 0x140009884b0}, {0x14000534a50, 0x2, 0x2?})
/usr/local/go/src/database/sql/ctxutil.go:82 +0x9c
database/sql.rowsiFromStatement({0x1051a4430, 0x106abf780}, {0x1051935e0, 0x1400073c900}, 0x1400095d640, {0x14000a91828, 0x2, 0x2})
/usr/local/go/src/database/sql/sql.go:2836 +0x118
database/sql.(*DB).queryDC(0x14000856a90?, {0x1051a4430, 0x106abf780}, {0x0, 0x0}, 0x14000d8abd0, 0x14000c71e40, {0x1043e2b1a, 0x23}, {0x14000a91828, ...})
/usr/local/go/src/database/sql/sql.go:1806 +0x26c
database/sql.(*DB).query(0x14000856a90, {0x1051a4430, 0x106abf780}, {0x1043e2b1a, 0x23}, {0x14000a91828, 0x2, 0x2}, 0x0?)
/usr/local/go/src/database/sql/sql.go:1754 +0xb4
database/sql.(*DB).QueryContext.func1(0xf8?)
/usr/local/go/src/database/sql/sql.go:1732 +0x54
database/sql.(*DB).retry(0x58?, 0x14000a91630)
/usr/local/go/src/database/sql/sql.go:1566 +0x4c
database/sql.(*DB).QueryContext(0x104e3c2e0?, {0x1051a4430?, 0x106abf780?}, {0x1043e2b1a?, 0x1400014af00?}, {0x14000a91828?, 0x14000a91718?, 0x104312878?})
/usr/local/go/src/database/sql/sql.go:1731 +0x94
database/sql.(*DB).Query(...)
/usr/local/go/src/database/sql/sql.go:1745
There's no way to react when Query|Exec
fails. Hooks interface should define an Error handler.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.