Python's sqlite3 + extensions
License: zlib License
Hi! I'm Anton, Golang developer and SQLite enthusiast. I work on open source projects (see below) and blog about programming.
Currently working on:
I've also written a book about the subset of SQL that helps with data analysis - SQL Window Functions Explained. It's a clear and visual introduction to the subject, taking the reader step-by-step from the basics to advanced topics.
Other (mini)books: Curl by example • Git by example • Grep by example
★ Subscribe to follow updates
I got this while running the test suite for my new sqlite-utils branch for sqlean:
https://github.com/simonw/sqlite-utils/tree/sqlean
I installed sqlean.py into the same environment and ran this:
pytest tests/test_gis.pyHere's the test failure:
============================================================================ test session starts =============================================================================
platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/simon/Dropbox/Development/sqlite-utils
plugins: hypothesis-6.68.2
collected 12 items
tests/test_gis.py .Fatal Python error: Bus error
Current thread 0x00000001e5735b40 (most recent call first):
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/python.py", line 1789 in runtest
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 339 in from_call
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 131 in runtestprotocol
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/main.py", line 324 in _main
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/main.py", line 270 in wrap_session
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/config/__init__.py", line 167 in main
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/_pytest/config/__init__.py", line 190 in console_main
File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/pytest", line 8 in <module>
Extension modules: sqlean._sqlite3 (total: 1)
zsh: bus error pytest tests/test_gis.py
This project is really cool! I've been using https://github.com/coleifer/pysqlite3 for a while, but I'm really excited to see this emerge as an alternative.
My primary interest is having a way to get the most recent version of SQLite installed on installations of Python that might otherwise be lagging behind. It looks like sqlean.py gives me SQLite 3.42.0 which is the most recent release!
I do have one problem: I'd like to use this with my Datasette tool, which is often deployed as a public site where anyone can execute SQL queries - e.g. https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list%28%29
It looks to me like sqlean.py may introduce some functions which are unsafe in that environment. These ones caught my eye:
eval() - can be used for select eval('create table tmp(value int)');define() - this looks like it modifies state as part of a SELECTregexp_like() and friends - do you know if these are susceptible to denial-of-service attacks? Some regular expression implementations are, others are not. https://observablehq.com/@asg017/introducing-sqlite-regex is an example of one that isn't.fileio_read(), fileio_write() etcWould it be possible to provide either a version of sqlean.py that only bundled "safe" functions (safe to be executed by untrusted users), or provide a mechanism whereby these unsafe functions could be disabled?
I'd love to update Datasette to be able to use sqlean.py if it's available, but I need to understand the security implications first.
Good day,
I'd like to request to add the new vector search extension:
https://github.com/asg017/sqlite-vss
I think it will make for a fine addition, that's all.
That is, publish a tar.gz sdist artifact to PyPI. Other packages that publish binary wheels also publish source distributions.
See https://packaging.python.org/en/latest/specifications/source-distribution-format/.
In terms of implementation, the cibuildwheel docs recommend using the build tool: https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/#github-actions
The only problem with that is it wouldn't include the contents of the sqlite directory since it's missing from the manifest file.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.