Code Monkey home page Code Monkey logo

queries's Introduction

Queries: PostgreSQL Simplified

Queries is a BSD licensed opinionated wrapper of the psycopg2 library for interacting with PostgreSQL.

The popular psycopg2 package is a full-featured python client. Unfortunately as a developer, you're often repeating the same steps to get started with your applications that use it. Queries aims to reduce the complexity of psycopg2 while adding additional features to make writing PostgreSQL client applications both fast and easy. Check out the Usage section below to see how easy it can be.

Key features include:

  • Simplified API
  • Support of Python 2.7+ and 3.4+
  • PyPy support via psycopg2cffi
  • Asynchronous support for Tornado
  • Connection information provided by URI
  • Query results delivered as a generator based iterators
  • Automatically registered data-type support for UUIDs, Unicode and Unicode Arrays
  • Ability to directly access psycopg2 connection and cursor objects
  • Internal connection pooling

Version Status Coverage License

Documentation

Documentation is available at https://queries.readthedocs.org

Installation

Queries is available via pypi and can be installed with easy_install or pip:

pip install queries

Usage

Queries provides a session based API for interacting with PostgreSQL. Simply pass in the URI of the PostgreSQL server to connect to when creating a session:

session = queries.Session("postgresql://postgres@localhost:5432/postgres")

Queries built-in connection pooling will re-use connections when possible, lowering the overhead of connecting and reconnecting.

When specifying a URI, if you omit the username and database name to connect with, Queries will use the current OS username for both. You can also omit the URI when connecting to connect to localhost on port 5432 as the current OS user, connecting to a database named for the current user. For example, if your username is fred and you omit the URI when issuing queries.query the URI that is constructed would be postgresql://fred@localhost:5432/fred.

If you'd rather use individual values for the connection, the queries.uri() method provides a quick and easy way to create a URI to pass into the various methods.

>>> queries.uri("server-name", 5432, "dbname", "user", "pass")
'postgresql://user:pass@server-name:5432/dbname'

Environment Variables

Currently Queries uses the following environment variables for tweaking various configuration values. The supported ones are:

  • QUERIES_MAX_POOL_SIZE - Modify the maximum size of the connection pool (default: 1)

Using the queries.Session class

To execute queries or call stored procedures, you start by creating an instance of the queries.Session class. It can act as a context manager, meaning you can use it with the with keyword and it will take care of cleaning up after itself. For more information on the with keyword and context managers, see PEP343.

In addition to both the queries.Session.query and queries.Session.callproc methods that are similar to the simple API methods, the queries.Session class provides access to the psycopg2 connection and cursor objects.

Using queries.Session.query

The following example shows how a queries.Session object can be used as a context manager to query the database table:

>>> import pprint
>>> import queries
>>>
>>> with queries.Session() as session:
...     for row in session.query('SELECT * FROM names'):
...         pprint.pprint(row)
...
{'id': 1, 'name': u'Jacob'}
{'id': 2, 'name': u'Mason'}
{'id': 3, 'name': u'Ethan'}

Using queries.Session.callproc

This example uses queries.Session.callproc to execute a stored procedure and then pretty-prints the single row results as a dictionary:

>>> import pprint
>>> import queries
>>> with queries.Session() as session:
...   results = session.callproc('chr', [65])
...   pprint.pprint(results.as_dict())
...
{'chr': u'A'}

Asynchronous Queries with Tornado

In addition to providing a Pythonic, synchronous client API for PostgreSQL, Queries provides a very similar asynchronous API for use with Tornado. The only major difference API difference between queries.TornadoSession and queries.Session is the TornadoSession.query and TornadoSession.callproc methods return the entire result set instead of acting as an iterator over the results. The following example uses TornadoSession.query in an asynchronous Tornado web application to send a JSON payload with the query result set.

from tornado import gen, ioloop, web
import queries

class MainHandler(web.RequestHandler):

    def initialize(self):
        self.session = queries.TornadoSession()

    @gen.coroutine
    def get(self):
        results = yield self.session.query('SELECT * FROM names')
        self.finish({'data': results.items()})
        results.free()

application = web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    ioloop.IOLoop.instance().start()

Inspiration

Queries is inspired by Kenneth Reitz's awesome work on requests.

History

Queries is a fork and enhancement of pgsql_wrapper, which can be found in the main GitHub repository of Queries as tags prior to version 1.2.0.

queries's People

Contributors

chrismcguire avatar dave-shawley avatar den-t avatar floftar avatar gmr avatar nvllsvm avatar orenitamar avatar tanveerg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

queries's Issues

Simple query fails if no rows found

Most simple example from docs:

for row in session.query('SELECT * FROM foo'):
    print row

Fails when there are no rows in the table:

Traceback (most recent call last):
  File "venv/lib/python3.9/site-packages/queries/results.py", line 45, in __iter__
    raise StopIteration
StopIteration

Expected result: for loop should just pass without errors

TornadoSession as a context manager

Hello ! First of all thank you for the work you put in your lib !

We spotted a strange behavior when using queries.TornadoSession in a context manager: The connection pool is not freed when exiting the with block. (This isn't the case for queries.Session).

It is true that the documentation encourages to use result.free() when working with asynchronous sessions but IMHO it would be a nice feature if both Session classes would behave the same.

Cheers !

Stacking tornado_session.Results.close calls doesn't close any connection

Maybe I am doing something wrong, but if I were to simply wait until later in the execution stack to call close on a tornado result, it never closes.

conn = queries.queries.TornadoSession(pool_max_size=3)
....
res1 = yield conn.result('some query')
res2 = yield conn.result('some query')
res3 = yield conn.result('some query')

res1.free()
res2.free()
res3.free()

res4 = yield conn.result('some query')
# PoolFullError: Pool <queries.pool.Pool object at $ID> is at its maximum capacity
res4.free()

If I were to free a result before using the connection again, things work fine. (I even tested with a counter by inspecting the conn_pool_manager._pools dict)

Am I missing something?

TornadoSession.validate enters endless loop upon failure

Not completely sure what is going on here, but when TornadoSession.validate is called with a bad password it fails with an exception which is fine except that something is left on the ioloop that eats the CPU alive. The following script will show the problem when you run it with a DSN that is valid except for the password:

#!/usr/bin/env python
#
import logging
import signal
import sys

import queries
from tornado import gen, ioloop


@gen.coroutine
def main():
    logger = logging.getLogger('main')
    for dsn in sys.argv[1:]:
        logger.info('starting session with dsn=%s', dsn)
        try:
            session = queries.TornadoSession(dsn)
            yield session.validate()
            logger.info('validated')
        except:
            logger.exception('FAILURE FOR %s', dsn)

    while True:
        yield gen.sleep(10)


def sig_handler(*args):
    logging.info('signal handler invoked with %r', args)
    iol = ioloop.IOLoop.instance()
    iol.add_callback_from_signal(iol.stop)


if __name__ == '__main__':
    if len(sys.argv) == 1:
        print('Usage:', sys.argv[0], 'DSN [DSN...]')
        sys.exit(1)

    logging.basicConfig(
        level=logging.DEBUG,
        format='[%(process)d] %(levelname)1.1s - %(name)s: %(message)s')
    signal.signal(signal.SIGINT, sig_handler)
    signal.signal(signal.SIGTERM, sig_handler)
    iol = ioloop.IOLoop.instance()
    iol.add_future(main(), lambda _: None)
    iol.start()

When given a valid DSN it will validate the session and sleep until you kill with Ctrl+C. Give it an invalid password and it will report and error and go into the same sleep cycle with something like the following log:

(env) Daves-MBP/queries-play% ./validate.py 'postgresql://postgres:[email protected]:32770/postgres'
[20562] I - main: starting session with dsn=postgresql://postgres:[email protected]:32770/postgres
[20562] D - queries.pool: Creating Pool: ea2750c53184e9ddec80659446cbe6fa (60/25)
[20562] D - queries.tornado_session: Creating a new connection for ea2750c53184e9ddec80659446cbe6fa
[20562] E - main: FAILURE FOR postgresql://postgres:[email protected]:32770/postgres
Traceback (most recent call last):
  File "./validate.py", line 18, in main
    yield session.validate()
  File "/Users/daveshawley/Source/python/queries-play/env/lib/python3.4/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/Users/daveshawley/Source/python/queries-play/env/lib/python3.4/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/daveshawley/Source/python/queries-play/env/lib/python3.4/site-packages/queries/tornado_session.py", line 460, in _poll_connection
    state = self._connections[fd].poll()
psycopg2.OperationalError: FATAL:  password authentication failed for user "postgres"

Now run top or similar process monitor and you'll notice that the process is consuming an entire core. It is not chewing through memory or stranding sockets (verified by lsof). It is simply looping very hard on something. I ran sudo dtrace -p 20562 -n 'syscall:::entry' to see what was going on and it looks like it might be failing to handle a socket error and retrying the connection sequence repeatedly. The dtrace output settles to:

  4    883           workq_kernreturn:entry
  2    941             write_nocancel:entry
  6    873                     kevent:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  0    333                     select:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry
  6    873                     kevent:entry
  2    941             write_nocancel:entry
  2    941             write_nocancel:entry

repeatedly. There are no calls to IO traps so I don't think that it is trying a new connection. It looks like it is simply looping repeatedly.

Support connection_factory?

I'll start with the question as I am not entirely certain if it is even possible; Is there a way to create a queries.Session using a connection_factory? And if not, would it be possible to add such support?

Or....
perhaps there is another way to accomplish the following...?

In cases where I wanted to use a cursor which supported both a NamedTuple result and also using a LoggingCursor I would combine the two cursors but I would also pass in the connection_factory which sets up the connection to utilize logging for the cursor. Is there some way of doing this in queries?

from psycopg2.extras import NamedTupleCursor, LoggingCursor, LoggingConnection


class MyLoggingCursor(LoggingCursor):
    def execute(self, query, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).execute(query, vars)

    def callproc(self, procname, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).callproc(procname, vars)


class MyLoggingConnection(LoggingConnection):
    def filter(self, msg, curs):
        duration = int((time.time() - curs.timestamp) * 1000)
        output = f"{msg}  ==> {curs.rowcount} rows, {duration:d} ms"
        return output

    def cursor(self, *args, **kwargs):
        kwargs.setdefault('cursor_factory', MixinLoggedNamedTupleCursor)
        return LoggingConnection.cursor(self, *args, **kwargs)


class MixinLoggedNamedTupleCursor(MyLoggingCursor, NamedTupleCursor):
    pass


db_conn = psycopg2.connect(host=db_host, port=db_port,
                           user=db_user, password=db_pass,
                           database=db_name, 
                           connect_timeout=timeout,
                           connection_factory=MyLoggingConnection
                           )
db_conn.initialize(logger)

python script cannot run by windows task scheduler with a simple "import queries"

Python script runs fine but not by windows task scheduler. It exits with (0x1) in task scheduler.
try the following script:

import logging
import pandas as pd
import sqlalchemy
import pprint
#import queries
from urllib.request import Request, urlopen
import datetime

logging.basicConfig(filename='./logs/mylog.log', filemode='a', format='%(asctime)s.%(msecs)03d %(name)s %(levelname)s %(message)s', datefmt='%Y-%m-%d,%H:%M:%S', level=logging.INFO)

logging.info("===== process started =====")

Argument 1 must be string, not None when creating TornadoSession

I'm trying to use TornadoSession in my project, but I can't make it work.
When the function TornadoSession() is called it raise the following exception:

 Exception in callback <functools.partial object at 0x7fb710f9c2b8>
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 600, in _run_callback
        ret = callback()
      File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
        return fn(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/queries/tornado_session.py", line 374, in on_connected
        cursor = self._get_cursor(conn)
      File "/usr/local/lib/python2.7/dist-packages/queries/session.py", line 311, in _get_cursor
        cursor_factory=self._cursor_factory)
    TypeError: argument 1 must be string, not None

It gives this error even with one of the example found in the documentation:

from tornado import gen, ioloop, web
import queries

class MainHandler(web.RequestHandler):

    def initialize(self):
        conf = queries.uri(host='localhost', port=5432, dbname="myDb", user="root", password="pass")
        self.session = queries.TornadoSession(conf)

    @gen.coroutine
    def get(self):
        results = yield self.session.query('SELECT * FROM myTable')
        self.finish({'data': results.items()})
        results.free()
application = web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    ioloop.IOLoop.instance().start()

What can cause this issue?

Is there an executemany using the TornadoSession?

I have seen the concurrent queries example by wrapping the sql statements in a list but that only works if you know beforehand how many queries you are going to send. Is there a method to executemany such as i would do below?

`cars = (
(1, 'Audi', 52642),
(2, 'Mercedes', 57127),
(3, 'Skoda', 9000),
(4, 'Volvo', 29000),
(5, 'Bentley', 350000),
(6, 'Citroen', 21000),
(7, 'Hummer', 41400),
(8, 'Volkswagen', 21600)
)

cur.executemany(query, cars)`

Thanks in advance!

psycopg2 as a dependency

Installing queries via pip install psycopg2 as a dependency. However on many platforms now psycopg2-binary package must be used because of lack of PG dev packages available. For instance, AWS Lambda, Azure Functions etc.

Therefore I believe that either psycopg2 should be removed as a direct dependency and let the user manually add the approprate package or offer a queries-binary variant.

Runing DML queries.

Can we update a table in database with this module? Like adding comments to table or changing a value in a column.

best,

It seems that Session doesn't accept uri to unix domain socket

It currently seemst to be impossible to connect to database via the unix domain socket.

The uri for socket-based connections (for default directory) looks like this: postgresql://user:pass@/database
And for custom location it looks like this: postgresql://user:pass@/database?host=/var/lib/pgsql

Add easy way to retrieve info about session state

I am using the TornadoSession with hundreds of queries and if somehow a result isn't freed, tracking it down is a nightmare. I put this together and was wondering if it would be useful to have in the core of the application (if there is a better way please let me know)

import queries


class MySession(queries.TornadoSession):

    def query(self, sql, parameters=None):
        self.sql = sql
        self.parameters = parameters

        return super(MySession, self).query(sql=sql, parameters=parameters)

    def info(self):
        info = []

        for pid, pool in self._pool_manager._pools.items():
            i = pool.idle_connections
            b = [pool.connections[k] for k in pool.connections
                 if pool.connections[k].busy]
            u = [pool.connections[k].used_by().sql for k in
                 pool.connections if pool.connections[k].busy]

            info.append({
                'pid': pid,
                'idle': len(i),
                'busy': len(b),
                'running_queries': u
            })

        return info

If you ever wanted info about the session, you'd just call session.info()

Password with %5D login failed

Hi,

I found that my password include %5D and it failed to login. And the same password I use psycopg2 library is OK.

It's some wrong my code or anything i ignored?

Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.