bradmontgomery / django-querycount Goto Github PK

Middleware that Prints the number of DB queries to the runserver console.

License: MIT License

Python 100.00%

database debugging developer-tools django python

django-querycount's Introduction

Hi there 👋

I've worked for years as a Software Engineer in various domains, spent a few more years as an Engineering Manager, and more recently as a Software Architect in the financial services industry.

📫 How to reach me: My personal email is my first name at my personal site.
😄 Pronouns: He/Him/His
🐍 Python is my go-to language of choice.
⚡ Fun fact: 💙🚴

django-querycount's People

Contributors

Stargazers

Watchers

django-querycount's Issues

WHERE_CLAUSE_REGEX bad escape problem

First of all: thanks so much for this module! I use it a shitload and it, basically, never lets me down. Appreciate it :)

I'm getting this little problemo with Django 4.2, Python 3.11, Psycopg 3.1.8:

Internal Server Error: /api/auth/login/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 136, in __call__
    response = self.process_response(request, response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/querycount/middleware.py", line 117, in process_response
    self._count_queries("response")
  File "/usr/local/lib/python3.11/site-packages/querycount/middleware.py", line 85, in _count_queries
    sql = self.WHERE_CLAUSE_REGEX.sub(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/re/__init__.py", line 317, in _subx
    template = _compile_repl(template, pattern)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/re/__init__.py", line 308, in _compile_repl
    return _parser.parse_template(repl, pattern)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/re/_parser.py", line 1078, in parse_template
    raise s.error('bad escape %s' % this, len(this)) from None
re.error: bad escape \x at position 55

Not exactly sure what's causing this, but thought it might be worth posting an issue about. Maybe data changed in some way between Django 4.1 and 4.2?

Include Python traceback to see where the query is called from

It's a very useful pluggable app to find duplicate queries / improve DB performance, thank you!

However, would it be possible to optionally include Python trace (or maybe even point to the ORM source line query), which caused repetitive SQL query in Django?

In case such feature would cause slowdown, it can be made optional.

Whitelist request patterns instead of blacklisting them

I often debug a single endpoint with a unique request pattern. To be able to only see that in the console I have to blacklist all other request patterns that my application may produce during debugging. That can be quite annoying. It would be helpful to have a separate whitelist setting like INCLUDE_REQUEST_PATTERNS where I just state what I want to be printed to the console and everything else is silenced. Of course the User would have to decide whether to blacklist or whitelist, as using both does not make much sense.

I can do a PR if you want, let me know if it's something worth putting time into.

How to view the total time of repeated sql queries

Can we show the total time take for the repeated calls individually

For example: presently it shows
Repeated 2 times.

Can we have
Repeated 2 times. [0.023]

"Repeated 1 times"

It prints this:

web_1            | Repeated 1 times.
web_1            | SELECT `django_session`.`session_key`,
web_1            | `django_session`.`session_data`, `django_session`.`expire_date` FROM
web_1            | `django_session` WHERE (`django_session`.`session_key` =
web_1            | 'cln2uzh2qk21qqb6sfg7nvsrqwg2crbm' AND `django_session`.`expire_date`
web_1            | > '2019-09-20 12:53:39.352699')

But I think it actually ever happens 1 time for that request.
So, the logic for this inclusion should be something like [query for queries if query.times > 1].

AttributeError: 'QueryCountMiddleware' object has no attribute 'get_response'

Test client throws above error when I try to test this snippet

class IndexPageTest(TestCase):
    def setUp(self):
        self.client = Client()

    def test_uses_login_template(self):
        response = self.client.get('/')
        self.assertTemplateUsed(response, 'index.html')

Although this code is perfectly working if I disable the middleware from settings.py .

Here is full output

Creating test database for alias 'default'...
E
======================================================================
ERROR: test_uses_login_template (....tests.IndexPageTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../tests.py", line 11, in test_uses_login_template
    response = self.client.get('/')
  File "....env/lib/python3.5/site-packages/django/test/client.py", line 529, in get
    **extra)
  File "....env/lib/python3.5/site-packages/django/test/client.py", line 333, in get
    return self.generic('GET', path, secure=secure, **r)
  File ".../.env/lib/python3.5/site-packages/django/test/client.py", line 409, in generic
    return self.request(**r)
  File ".../.env/lib/python3.5/site-packages/django/test/client.py", line 494, in request
    six.reraise(*exc_info)
  File "....env/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
    raise value
  File ".../.env/lib/python3.5/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
  File "....env/lib/python3.5/site-packages/django/utils/deprecation.py", line 136, in __call__
    response = self.get_response(request)
AttributeError: 'QueryCountMiddleware' object has no attribute 'get_response'

----------------------------------------------------------------------
Ran 1 test in 2.621s

FAILED (errors=1)

Using sql parse and pygments to show sql formatted nicely

I wanted the sql to be formatted so i have tried the below code for that. Can you incorporate sqlparse also if possible

def _duplicate_queries(self, output):
        """Appends the most common duplicate queries to the given output."""
        if QC_SETTINGS['DISPLAY_DUPLICATES']:
            for query, count in self.queries.most_common(QC_SETTINGS['DISPLAY_DUPLICATES']):
                lines = '\nRepeated {0} times.'.format(count)
                #lines += wrap(query)
                #lines = "\n".join(lines) + "\n"
                lines += "\n" + self._str_query(query) + "\n"
                output += self._colorize(lines, count)
        return output

    def _str_query(self,sql):

        # Check if Pygments is available for coloring
        try:
            import pygments
            from pygments.lexers import SqlLexer
            from pygments.formatters import TerminalTrueColorFormatter
        except ImportError:
            pygments = None
        # Check if sqlparse is available for indentation
        try:
            import sqlparse
        except ImportError:
            sqlparse = None
        # Remove leading and trailing whitespaces
        if sqlparse:
            # Indent the SQL query
            sql = sqlparse.format(sql, reindent=True)
        if pygments:
            # Highlight the SQL query
            sql = pygments.highlight(
                sql,
                SqlLexer(),
                #TerminalTrueColorFormatter(style='monokai')
                TerminalTrueColorFormatter()
            )

        return sql

celery tasks, in DEBUG I want them to print the query count in the same manner middleware does
telegram bots, we do develop some telegram bots on top of django. telegram bots serve as clients which does not have any django's request instance. But I still want this app to work

Thanks!

Proper documentation

We've probably out-grown the readme-only documentation.

Docs in a docs/ directory
GHA to build them when merged.
hosted on github pages.

Disable colors when not supported (or explicitely disabled)

On Windows, using powershell, the output looks bad as colors are not supported :

http://127.0.0.1:8000/admin/jsi18n/
?[0m|------|-----------|----------|----------|----------|------------|
| Type | Database  |   Reads  |  Writes  |  Totals  | Duplicates |
|------|-----------|----------|----------|----------|------------|
?[32m| RESP |  default  |    2     |    0     |    2     |     1      |
?[0m|------|-----------|----------|----------|----------|------------|
?[32mTotal queries: 2 in 0.0160s

Colors should be disabled when not supported or disabled. I believe Django has all the logic needed in django\core\management\color.py

Support for Django 4+

This project hasn't currently been tested against django 4.

bradmontgomery / django-querycount Goto Github PK

django-querycount's Introduction

Hi there 👋

django-querycount's People

Contributors

Stargazers

Watchers

Forkers

django-querycount's Issues

Recommend Projects

Recommend Topics

Recommend Org