packaging dqlite in linux distribution / static linking to sqlite3 fork about dqlite HOT 17 CLOSED

@spumer, yes it's coming hopefully in the next few weeks. There are a few boring symbol renames that I have todo in order to do that.

from dqlite.

Replacing is not a good option (as fork is often older than official upstream for example; problematic dependencies specification (as package X needs forked version; package Y doesn't care; package Z want's something newer than fork can provide, so only upstream package works etc). Didn't look into diff, yet but I could go with patching official salite3 route if it's not invasive.

Anyway this issue is meant to be about generic solution, not only for me.

from dqlite.

@coleifer a couple of things: 1) you can compile SQLite with any flag and extension and be confident that the dqlite patch will still work fine 2) yes, amalgamation and static linking support is on my TODO list, but will do it as time permits. Contributions are welcome.

from dqlite.

Hi @arekm,

regarding proposing changes upstream to SQLite, that's my long term plan (6 months/ one year). Since SQLite's upstream is extremely conservative, I want to be absolutely confident about the patch, and also have a good real-world usage track record of it.

In the meantime, one possible solution that I've been discussing with a Debian developer is to do something in between packaging fork and statically linking: you'd include the forked sqlite3 .so object in the dqlite package but under a different name than the non-forked one, and link dqlite against it.

I assume that it might require some patch by the packager, but would that be acceptable?

On the other hand, I understand the convenience that statically linking would bring, so I'll try to think about the best way to do it.

from dqlite.

The easiest for everyone is change shared library name in sqlite3 fork and make dqlite use that new name when linking. Here I would package sqlite3 fork as separate package (instead of including sqlite3new-name.so in dqlite package itself).

So for example:

sqlite3dq-version-release.{deb,rpm} with libsqlitedq.so.X.Y
sqlite3dq-devel-version-release.{deb,rpm} with include/sqlite3dq.h
dqlite linking do libsqlitedq.so...

Using shared forked sqlite3(dq).so can lead to problems though if some app will link to official sqlite3.so and to dqlite.so linked to forked sqlite3dq.so - symbol clashes etc but at this moment this is unlikely as not much software uses dqlite.so.

from dqlite.

@arekm unfortunately that's not so unlikely, since the go-sqlite3 Go package links to non-forked sqlite3.so and it's typically used to together with dqlite (e.g. LXD uses dqlite for its cluster database and go-sqlite3 for its local database)

from dqlite.

Any shared linking to fork will be problematic and only static linking makes sense then (until upstreamed).

from dqlite.

I'll check how static linking behaves when also dynamically linking to the official libsqlite3.so in a Go program that uses go-sqlite3.

Some distros seem to have successfully managed to package dqlite using a package which sports the forked libsqlite3.so (without any change in the .so name), using the trick that such package must conflict/replace the official sqlite package. For example see the ArchLinux sqlite-replication package here:

https://aur.archlinux.org/packages/sqlite-replication/

So that could be done for RPM too. Since the forked sqlite is just extending the official one and not changing its behavior in a way unless you use the replication features, all applications working with the official sqlite will also work with the forked one.

Would such approach be acceptable in your case?

I'll try to work on the statically linked forked sqlite approach too, but it will be low-priority for now.

from dqlite.

So the fork should not lag with respect to the official upstream. I regularly merge upstream releases into the fork. If I'm late, please ping me.

I understand that you're looking for a generic solution, and I agree that it's ultimately needed. However I think that a bit of diligence in keeping the fork up-to-date with the latest version is enough to rule out the case of "package Z wants something newer than the fork can provide".

If you are in the position of just applying the patch to the official sqlite package for your distro, that'd be great. It's really just adding some hooks, and as I mentioned there's absolutely no behavior change for applications not using those hooks. In the last year there were only trivial conflicts when merging new upstream releases to the fork.

In the longer term I do plan to submit the patch upstream as mentioned. I'll also try to work on the static linking approach, but it will take some time since it's low-priority at the moment. I'm happy to accept patches tho :)

from dqlite.

I wonder if github allows to get patch easily. This not working for me:

https://github.com/mackyle/sqlite/compare/version-3.25.2...CanonicalLtd:version-3.25.2+replication3.patch

(doesn't apply to 3.25.2, patch contains too much). Ideas?

from dqlite.

I'm not sure if there's an easy way to do that via GitHub. For good measure, I've added the diff as asset to the release that gets built by Travis when I push a new tag.

I've just merged into the fork the 3.25.3 version that upstream has released yesterday, and the associated diff is here:

https://github.com/CanonicalLtd/sqlite/releases/download/version-3.25.3%2Breplication3/sqlite-3.25.3.diff

Hope it helps.

from dqlite.

The entry point for all assets is https://github.com/CanonicalLtd/sqlite/releases.

from dqlite.

Thanks, patch working great (so used "patched upstream sqlite3" solution).

from dqlite.

Hi!

Now that buster is released, I can finally start having a look at lxd (and so dqlite) again.

I'll check how static linking behaves when also dynamically linking to the official libsqlite3.so in a Go program that uses go-sqlite3.

Did you check after all ? That would probably make things a lot simpler for packages maintainers.

I'll try to work on the statically linked forked sqlite approach too, but it will be low-priority for now.
Mhmm, I'm not sure, are you talking here about the same thing or not ?

How about upstreaming the patches? Anything new on this front?

Cheers,

from dqlite.

I have applications that depend on Sqlite being compiled with certain extensions, etc. The last thing I want to do is need to stomp on my system sqlite with a forked version with potentially sketchy changes -- especially as I'm just evaluating dqlite and want to develop against it.

Statically linking sqlite should be pretty straightforward -- just generate the amalgamation files (sqlite3.c and sqlite3.h) and compile them alongside the dqlite source. Renaming the shared library seems like such a kludge, especially when sqlite3 has gone to such efforts to make it easy to embed.

from dqlite.

Yea, static linking will be great feature. It used in sqlite extension for Sourcemod
Amalgamated sources places in subfolder sqlite-source.

I want use dqlite as DB for shared data between my gameservers (based on Sourcemod).

Any plans @freeekanayaka ?

from dqlite.

We no longer use a sqlite3 fork.

from dqlite.