blopker / djdt-flamegraph Goto Github PK

View Code? Open in Web Editor NEW

186.0 186.0 14.0 415 KB

Flamegraphs for Django Debug Toolbar

License: MIT License

Makefile 2.58% Python 20.52% Perl 76.65% HTML 0.25%

djdt-flamegraph's People

Contributors

Stargazers

Watchers

Forkers

graingert mgedmin quantus ruandao ljodal ianzapolsky hyreas pktiuk theilgaard dwafwdw

djdt-flamegraph's Issues

Signal handling may be delayed, which leads to dropped samples

I've previously mentioned this issue in the comments of PR #1, but now that I understand what the problem is, I think it's worth creating a separate issue, not tied to any particular PR.

djdt-flamegraph uses interval timers that periodically generate signals, and registers a signal handler to sample the current stack frame. Now, the way CPython implements signal handlers is that they set an internal flag, which is checked the next time the CPython interpreter enters the main eval loop.

A consequence of this is that if the main thread is blocked in some C code, signal handling gets delayed for an unbounded time, and you're getting a skewed profile picture because you're missing a significant number of samples. Example: executing an SQL query via psycopg2 may delay stack sampling for hundreds or thousands of milliseconds.

ValueError: signal only works in main thread

Running Django 3.0 on Python 3.8, using python ./manage.py runserver

  File "/usr/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 181, in process_request
    return self.get_response(request)
  File "/usr/lib/python3.8/site-packages/djdt_flamegraph/djdt_flamegraph.py", line 63, in process_request
    self.sampler.start()
  File "/usr/lib/python3.8/site-packages/djdt_flamegraph/djdt_flamegraph.py", line 90, in start
    signal.signal(signal.SIGALRM, self._sample)
  File "/usr/lib64/python3.8/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

Signals + subprocesses may lead to livelock

When you're profiling a view that uses subprocess.Popen(), it might livelock:

the clone() syscall (used to implement os.fork() under the hood) may take 2-3 milliseconds on my machine (as measured by strace -T while the flamegraph was off)
a SIGALRM arrives every 1 ms and interrupts the clone()
clone() returns -EINTR
the signal handler runs
clone() is then restarted
go to step 1

This is very clearly visible in strace output. Externally this is visible as a Django process eating 100% CPU and sometimes not making any progress.

Sometimes the view would actually finish, after an extra 10-20 seconds of this processing, which shows in the flamegraph as a call stack terminating in subprocess.Popen._wait().

I don't know if this can be fixed.

Possible mitigations: don't use subprocess.Popen() from the main thread, use a lower sampling frequency for the SIGALRM timer.

make example returns error

I am not able to run example project

$ make example
virtualenv --python=python example/env
created virtual environment CPython3.8.10.final.0-64 in 174ms
  creator CPython3Posix(dest=/tmp/tmp/djdt-flamegraph/example/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/pawel/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
example/env/bin/pip install -r example/requirements.txt
Collecting django
  Using cached Django-4.0-py3-none-any.whl (8.0 MB)
Collecting django-debug-toolbar
  Using cached django_debug_toolbar-3.2.2-py3-none-any.whl (200 kB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.2-py3-none-any.whl (42 kB)
Collecting asgiref<4,>=3.4.1
  Using cached asgiref-3.4.1-py3-none-any.whl (25 kB)
Collecting backports.zoneinfo; python_version < "3.9"
  Using cached backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB)
Installing collected packages: sqlparse, asgiref, backports.zoneinfo, django, django-debug-toolbar
Successfully installed asgiref-3.4.1 backports.zoneinfo-0.2.1 django-4.0 django-debug-toolbar-3.2.2 sqlparse-0.4.2
example/env/bin/pip install -e `pwd`
Obtaining file:///tmp/tmp/djdt-flamegraph
Installing collected packages: djdt-flamegraph
  Running setup.py develop for djdt-flamegraph
Successfully installed djdt-flamegraph
example/env/bin/python example/manage.py migrate
Traceback (most recent call last):
  File "example/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/__init__.py", line 425, in execute_from_command_line
    utility.execute()
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/base.py", line 373, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 75, in handle
    self.check(databases=[database])
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/management/base.py", line 438, in check
    all_issues = checks.run_checks(
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/checks/registry.py", line 77, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/urls/resolvers.py", line 446, in check
    for pattern in self.url_patterns:
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/urls/resolvers.py", line 632, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/urls/resolvers.py", line 625, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/tmp/djdt-flamegraph/example/mysite/urls.py", line 16, in <module>
    from django.conf.urls import url, include, patterns
ImportError: cannot import name 'url' from 'django.conf.urls' (/tmp/tmp/djdt-flamegraph/example/env/lib/python3.8/site-packages/django/conf/urls/__init__.py)
make: *** [Makefile:21: example/env] Błąd 1

AttributeError: 'NoneType' object has no attribute 'get' when this package is installed.

Installing the flame graph panel causes an AttributeError.

AttributeError
Exception Value: 'NoneType' object has no attribute 'get'

File "…/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)
File "…/debug_toolbar/middleware.py" in __call__
  69.         content_encoding = response.get("Content-Encoding", "")

According to the documentation, process_request should return a response which it doesn't look like this panel is doing.

500: INTERNAL SERVER ERROR

I get the Flamegraph tab in the Toolbar, but I get a: 500: INTERNAL SERVER ERROR

Details:
Traceback (most recent call last):
File "/home/usr/.pyenv/versions/pq/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/home/usr/.pyenv/versions/3.4.3/lib/python3.4/contextlib.py", line 30, in inner
return func(_args, *_kwds)
File "/home/usr/.pyenv/versions/pq/lib/python3.4/site-packages/debug_toolbar/views.py", line 19, in render_panel
content = panel.content
File "/home/usr/.pyenv/versions/pq/lib/python3.4/site-packages/djdt_flamegraph/djdt_flamegraph.py", line 55, in content
'flamegraph': flamegraph.stats_to_svg(self.sampler.get_stats())
File "/home/usr/.pyenv/versions/pq/lib/python3.4/site-packages/djdt_flamegraph/flamegraph.py", line 15, in stats_to_svg
out, _ = proc.communicate(stats)
File "/home/usr/.pyenv/versions/3.4.3/lib/python3.4/subprocess.py", line 960, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/home/usr/.pyenv/versions/3.4.3/lib/python3.4/subprocess.py", line 1602, in _communicate
input_view = memoryview(self._input)
TypeError: memoryview: str object does not have the buffer interface

nicer error page for missing start parameters

Did someone already try to make a nicer error message for the "signal only works in main thread" error?

If its possible id like it to just remind to add "--nothreading --noreload" before activating this djdt panel. I offer myself for implementing this, but as i have just a bare knowledge about this threading stuff, i better ask before running into dead ends. :)

blopker / djdt-flamegraph Goto Github PK

djdt-flamegraph's People

Contributors

Stargazers

Watchers

Forkers

djdt-flamegraph's Issues

Signal handling may be delayed, which leads to dropped samples

ValueError: signal only works in main thread

Signals + subprocesses may lead to livelock

make example returns error

AttributeError: 'NoneType' object has no attribute 'get' when this package is installed.

500: INTERNAL SERVER ERROR

nicer error page for missing start parameters

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent