panic in sqlite about dqlite HOT 14 CLOSED

I'm not entirely familiar with the internals of the Go build toolchain, but could it be that the pkg object for github.com/mattn/go-sqlite3 was compiled without -tags libsqlite3? I'd first exclude any weird compile/linking issue by wiping the Go build cache (probably ~/.cache/go-build and $GOPATH/pkg).

from dqlite.

Another thing you might want to try is export GO_DQLITE_MULTITHREAD=1 (or whatever other means to set that environment variable in the Go process whose source uses both go-dqlite and go-sqlite3). Since dqlite itself does not need SQLite to use any thread locking, it disables them by default, however it could be a problem if the same process also uses SQLite directly and does need locking.

from dqlite.

Yes, it was compiled with libsqlite3. It doesn't always panic, it is fairly random when it does. Super frustrating because I can't reproduce it on demand. But right now it's blocking me being able to merge dqlite into master for k3s because it seems to break non-dqlite. I'm going to try to bisect changes.

from dqlite.

@freeekanayaka are you aware of anybody that has a apline package of libsqite+replication?

from dqlite.

I'm not. If there is, that's probably because somebody has been trying to package LXD for alpine.

from dqlite.

What's the panic message that you get? I don't think you pasted it above.

from dqlite.

@freeekanayaka Is the lower case t here intentional? https://github.com/canonical/sqlite/blob/master/src/pager.c#L4949 #ifdef SQLITE_ENABLE_WAL_REPLICAtION

from dqlite.

No :/

from dqlite.

This is the full panic

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x34ee9e3]

runtime stack:
runtime.throw(0x3f10401, 0x2a)
	/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x4a9

goroutine 348856 [syscall]:
runtime.cgocall(0x3416910, 0xc004df32b0, 0x0)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc004df3280 sp=0xc004df3248 pc=0x40451b
github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3._Cfunc_sqlite3_close_v2(0x85e5ec8, 0x0)
	_cgo_gotypes.go:607 +0x49 fp=0xc004df32b0 sp=0xc004df3280 pc=0xf280a9
github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3.(*SQLiteConn).Close.func1(0xc008469aa0, 0x0)
	/go/src/github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3/sqlite3.go:1646 +0x5f fp=0xc004df32f0 sp=0xc004df32b0 pc=0xf3ebef
github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3.(*SQLiteConn).Close(0xc008469aa0, 0x8, 0xc0122dab00)
	/go/src/github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3/sqlite3.go:1646 +0x2f fp=0xc004df3320 sp=0xc004df32f0 pc=0xf37d8f
database/sql.(*driverConn).finalClose.func2()
	/usr/local/go/src/database/sql/sql.go:521 +0x49 fp=0xc004df3358 sp=0xc004df3320 pc=0xf192a9
database/sql.withLock(0x48713e0, 0xc00eed7300, 0xc004df33f0)
	/usr/local/go/src/database/sql/sql.go:3097 +0x63 fp=0xc004df3380 sp=0xc004df3358 pc=0xf18f23
database/sql.(*driverConn).finalClose(0xc00eed7300, 0x3e1d580, 0x7f925ec3bcb0)
	/usr/local/go/src/database/sql/sql.go:519 +0x130 fp=0xc004df3418 sp=0xc004df3380 pc=0xf0c7c0
database/sql.finalCloser.finalClose-fm(0xc0008e00e0, 0x4828b40)
	/usr/local/go/src/database/sql/sql.go:565 +0x2f fp=0xc004df3440 sp=0xc004df3418 pc=0xf1b98f
database/sql.(*driverConn).Close(0xc00eed7300, 0xc00eed7300, 0x0)
	/usr/local/go/src/database/sql/sql.go:500 +0x138 fp=0xc004df3490 sp=0xc004df3440 pc=0xf0c668
database/sql.(*DB).putConn(0xc0008e00c0, 0xc00eed7300, 0x0, 0x0, 0xc0000ae200)
	/usr/local/go/src/database/sql/sql.go:1277 +0x1c8 fp=0xc004df3500 sp=0xc004df3490 pc=0xf0ffb8
database/sql.(*driverConn).releaseConn(...)
	/usr/local/go/src/database/sql/sql.go:421
database/sql.(*driverConn).releaseConn-fm(0x0, 0x0)
	/usr/local/go/src/database/sql/sql.go:420 +0x4c fp=0xc004df3538 sp=0xc004df3500 pc=0xf1ba0c
database/sql.(*Rows).close(0xc00efc3200, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/database/sql/sql.go:3001 +0x15a fp=0xc004df3588 sp=0xc004df3538 pc=0xf189fa
database/sql.(*Rows).Close(...)
	/usr/local/go/src/database/sql/sql.go:2972
database/sql.(*Row).Scan(0xc00975d120, 0xc004df3628, 0x1, 0x1, 0x0, 0x0)
	/usr/local/go/src/database/sql/sql.go:3053 +0x18c fp=0xc004df35c8 sp=0xc004df3588 pc=0xf18c5c
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/drivers/generic.(*Generic).CurrentRevision(0xc0011a6000, 0x48daca0, 0xc011d2dbf0, 0x0, 0x0, 0x0)

From my best guess, it's trying to close an already closed connection.

from dqlite.

Oh, I just finally understood what you were saying about GO_DQLITE_MULTITHREAD. Hopefully that's my issue. Will try.

from dqlite.

Setting GO_DQLITE_MULTITHREAD=1 seemed to help a lot. It's a bit hard to ensure that env var is set, so right now I'm just forking go-dqlite and dropping that init code. After setting that, I'm not getting that same random panic, but I did encounter "database disk image is malformed" which I've never seen before. I'm now trying to get to the root of that.

from dqlite.

@ibuildthecloud yeah GO_DQLITE_MULTITHREAD is a bit of a pity: it's not intended to be a public API, at the moment it's more a hack. The problem is that locking settings like this are global in SQLite, so I can't disable them just for dqlite, and even worse if you want to change those settings at all you need to effectively do it before any SQLite API is used (e.g. sqlite3_open), otherwise they get lazily initialized with default values and you can't change that afterwise. So it's not something that I can expose in dqlite.New() or anything like that, it somehow needs to happen a module's init() function in order to take effect.

from dqlite.

@freeekanayaka Is the lower case t here intentional? https://github.com/canonical/sqlite/blob/master/src/pager.c#L4949 #ifdef SQLITE_ENABLE_WAL_REPLICAtION

I fixed this and tagged a new version-3.30.1+replication4 release. The deb packages have already propagated to the devel and stable PPAs.

from dqlite.

Things are work well now that I set GO_DQLITE_MULTITHREAD=1. I opened canonical/go-dqlite#54 to request an API or something be added, because I am currently forking go-dqlite to set the value properly.

from dqlite.