jazzband / django-celery-monitor Goto Github PK
View Code? Open in Web Editor NEWCelery Monitoring for Django
Home Page: https://django-celery-monitor.readthedocs.io/
License: Other
Celery Monitoring for Django
Home Page: https://django-celery-monitor.readthedocs.io/
License: Other
We have many of this errors and produced in this code:
def update_heartbeat(self, hostname, heartbeat, update_freq):
with transaction.atomic():
# check if there was an update in the last n seconds?
interval = Now() - timedelta(seconds=update_freq)
recent_worker_updates = self.filter(
hostname=hostname,
last_update__gte=interval,
)
if recent_worker_updates.exists():
# if yes, get the latest update and move on
obj = recent_worker_updates.get()
else:
# if no, update the worker state and move on
obj, _ = self.select_for_update_or_create(
hostname=hostname,
defaults={'last_heartbeat': heartbeat},
)
return obj
We have identified the bug in interval line.
Interval is generated in dynamic way (celery_monitor_workerstate.last_update >= ((CURRENT_TIMESTAMP - INTERVAL '60.000000' SECOND_MICROSECOND)))
and the query results from recent_worker_updates.exists()
and obj = recent_worker_updates.get()
is different.
With a fixed interval value, we can avoid this bug.
Currently we're using Django's automatic transaction management when we call get_or_create
or update_or_create
. We should in addition use select_for_update
to make sure that the data we're writing to the TaskState and WorkerState models work as intended.
This is especially true if the Camera is deployed in a way that would have multiple instances run side by side for scalability reasons.
It would be nice to have example scripts, similar to Celery itself, for upstart/systemd/etc. While I'm sure I can just copy and modify the celerybeat scripts for the purpose, I would be more confident if I could bring something over directly from this project.
Hello, it is said that we can control a few things:
Is this implemented ? Nothing works and does not appear in the source code ?
I have set them as followed in the settings.py file and didn't work at all...
monitor_task_success_expires = timedelta(seconds=10) # State: SUCCESS
monitor_task_error_expires = timedelta(days=7) # State: RETRY, FAILURE, REVOKED
monitor_task_pending_expires = timedelta(days=7) # State: PENDING, RECEIVED, STARTED, REJECTED, RETRY
Could you provide an example of where to set this ?
Thanks a lot,
Jonathan
hi @jezdez i just realized that you are not "storing" the celery beat scheduled tasks. The old django-celery integration allowed this so we could keep track if the beat was running properly. so i was wondering if that was intentional or not.
I remember a while ago i asked for something similar here celery/django-celery-results#16 but maybe this might make sense in this project.
My recommendation was to save it to cache memory first and eventually dumping it in the database (could be the same table or a different table).
What do you think?
This should move the time value to the interval field, so it's more in line with cron-like UX patterns, and the date of the first iteration kept in the start_date field.
Same thing for the end date and indicate if it's inclusive or exclusive.
This issue tracks the implementation of the Jazzband guidelines for the project django-celery-monitor
It was initiated by @jezdez who was automatically assigned in addition to the Jazzband roadies.
See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.
Feel free to ping a Jazzband roadie if you have any question.
README
fileCONTRIBUTING.md
or CONTRIBUTING.rst
filejazzband
account to PyPI project as maintainer role (e.g. URL: https://pypi.org/manage/project/django-celery-monitor/collaboration/)jazzband-bot
as maintainer to the Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/django-celery-monitor/users/)Description | Celery Monitoring for Django |
Homepage | https://django-celery-monitor.readthedocs.io/ |
Stargazers | 101 |
Open issues | 25 |
Forks | 42 |
Default branch | master |
Is a fork | False |
Has Wiki | False |
Has Pages | False |
@jezdez i am running into this error:
celerycam_1 | -> evcam: Taking snapshots with django_celery_monitor.camera.Camera (every 2.0 secs.)
celerycam_1 | Traceback (most recent call last):
celerycam_1 | File "/usr/local/bin/celery", line 11, in <module>
celerycam_1 | sys.exit(main())
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/__main__.py", line 14, in main
celerycam_1 | _main()
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 326, in main
celerycam_1 | cmd.execute_from_commandline(argv)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 488, in execute_from_commandline
celerycam_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 281, in execute_from_commandline
celerycam_1 | return self.handle_argv(self.prog_name, argv[1:])
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 480, in handle_argv
celerycam_1 | return self.execute(command, argv)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/celery.py", line 412, in execute
celerycam_1 | ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 285, in run_from_argv
celerycam_1 | sys.argv if argv is None else argv, command)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 368, in handle_argv
celerycam_1 | return self(*args, **options)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/base.py", line 244, in __call__
celerycam_1 | ret = self.run(*args, **kwargs)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/events.py", line 118, in run
celerycam_1 | detach=detach)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/bin/events.py", line 144, in run_evcam
celerycam_1 | return cam()
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/events/snapshot.py", line 101, in evcam
celerycam_1 | maxrate=maxrate, timer=timer)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/celery/utils/imports.py", line 53, in instantiate
celerycam_1 | return symbol_by_name(name)(*args, **kwargs)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/kombu/utils/imports.py", line 56, in symbol_by_name
celerycam_1 | module = imp(module_name, package=package, **kwargs)
celerycam_1 | File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
celerycam_1 | return _bootstrap._gcd_import(name[level:], package, level)
celerycam_1 | File "<frozen importlib._bootstrap>", line 986, in _gcd_import
celerycam_1 | File "<frozen importlib._bootstrap>", line 969, in _find_and_load
celerycam_1 | File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
celerycam_1 | File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
celerycam_1 | File "<frozen importlib._bootstrap_external>", line 665, in exec_module
celerycam_1 | File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/django_celery_monitor/camera.py", line 14, in <module>
celerycam_1 | from .models import WorkerState, TaskState
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/django_celery_monitor/models.py", line 21, in <module>
celerycam_1 | class WorkerState(models.Model):
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py", line 105, in __new__
celerycam_1 | app_config = apps.get_containing_app_config(module)
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 237, in get_containing_app_config
celerycam_1 | self.check_apps_ready()
celerycam_1 | File "/usr/local/lib/python3.5/dist-packages/django/apps/registry.py", line 124, in check_apps_ready
celerycam_1 | raise AppRegistryNotReady("Apps aren't loaded yet.")
celerycam_1 | django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
I am using:
Django==1.10.7
celery==4.0.2
django-celery-results==1.0.1
django-celery-beat==1.0.1
django-celery-monitor==1.0.1
anyjson==0.3.3
Any ideas why i am getting this or how to fix it??
THanks
What will this package actually deliver? Extra commands? A webpage? An alternative to flower?
Hey @jezdez
Does django-celery-monitor conflict with django-celery-results. I have a couple issues but I'm wondering if these two results
are effectively stepping on each other.
Would you mind sharing how you have this setup and if you are using periodic tasks?
Thanks
Hi,
I am using django==1.8.19, celery==4.1.1,django-celery-monitor==1.1.2. And I want celery tasks to be preserved a bit longer in celery monitor. But, tasks stays only for 5 days max. I tried making configuration as stated in the readme. But, no use. Here is my code.
from datetime import timedelta
monitor_task_success_expires = timedelta(days=6*30)
monitor_task_error_expires = timedelta(days=6*30)
monitor_task_pending_expires = timedelta(days=6*30)`
All other settings in my django settings file are capital letters. do I have make these also in capital letters? like this?
MONITOR_TASK_SUCCESS_EXPIRES = timedelta(days=6*30)
please help me.
Do you have any tips for migrating from celery 3.1.13 -> 4.x and django-celery 3.1.10 to [list the successor packages]? I'm in the process of migrating an old Django 1.6 site to Django 1.8. At the moment everything runs well under Django 1.8 with celery 3.1.25 and django-celery 3.2.1, but I want to migrate it a little bit further, keeping current functionality intact as much as possible.
Several parts of django-celery are currently in use (admin):
Any advice is highly appreciated. Thanks in advance!
Your master build is failing, presumably due to the HTTP 503 error in https://travis-ci.org/jezdez/django-celery-monitor/jobs/324656984. It looks like a temporary network issue, so rerunning the build might do the trick.
Can you guys rerun the build?
Hi!
I am having an issue with some tasks that once they have successfuly finished do not update on admin panel, so they freeze with started or retry status.
I run celery monitor with:
celery -A proj events -l info --camera django_celery_monitor.camera.Camera
I didn't find anything in the logs for now.
Dependabot can't evaluate your Python dependency files.
As a result, Dependabot couldn't check whether any of your dependencies are out-of-date.
The error Dependabot encountered was:
Illformed requirement [">dev"]
So currently HTML in the admin is not supported and HTML tags get rendered. Needs to be refactored to use format_html.
https://docs.djangoproject.com/en/2.0/ref/utils/#django.utils.html.format_html
Hello,
I have this error:
ImportError: cannot import name 'render_to_response' from 'django.shortcuts'
when will it be compatible with django 3 and celery 5?
regards
Matias
Hi,
I can't visualize meta information like root id, parent id, ...
This is normal?
With django_celery_results I have this informations
Thank
Is it possible to specify a name of a database that contains tables for the celery monitor? In my project it is not 'default'. My 'default' is empty.
Hello.
I am getting a quite high CPU usage from the celery monitor process, around 15% constantly, I'm not having a big load or queue so I can't really explain that (I process maybe like maximum 5 tasks per seconds in the worst case).
Do you have the same behaviour ? How can I troubleshoot that more ?
Here is my monitor command
pipenv run celery -A website_manager_service events -l info --camera django_celery_monitor.camera.Camera --frequency=2.0 --pidfile=
Thanks
In the docs it is stated that the name of the variables for the timedelta of expiration is
but in the code it appears to me that the names are different
Hello
I don't know you but I think it should be interesting to set the color of the STARTED string state in the admin to something darker, it's quite difficult to read this flashy yellow on a white background :)
Thanks
When I use stopwait
to stop running workers, it waits correctly, but the last task to run does not get its status updated to SUCCESS
, it remains STARTED
, even when workers and camera are restarted.
Hi there,
I'm using the following decorator for one of my tasks:
@app.task(autoretry_for=(Exception,))
I was hoping that the number of retries would be reflected on the "retries" column under table celery_monitor_taskstate but for some reason, it's always 0. I'm positive that the task is failing and retrying since I can see the state changes via Flower.
Any ideas? Is there anything I'm missing?
can be closed, false positive
I see that django-celery-monitor recently became a Jazzband project, but doesn't look like it's maintained.
It doesn't even work with Django 3.0 (probably even 2.2, judging by the issues).
I just wanted to check if there are plans on activating the project or is it abandoned?
Thanks!
Celery Monitor no longer works with Django 2.2:
More specifically:
[2019-04-05 10:05:57,937: ERROR/MainProcess] Error in timer: TypeError('get() argument after ** must be a mapping, not str')
Traceback (most recent call last):
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 166, in apply_entry
entry()
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
return self.fun(*self.args, **self.kwargs)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 127, in _reschedules
return fun(*args, **kwargs)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/snapshot.py", line 72, in capture
self.state.freeze_while(self.shutter, clear_after=self.clear_after)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/state.py", line 456, in freeze_while
return fun(*args, **kwargs)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/celery/events/snapshot.py", line 69, in shutter
self.on_shutter(self.state)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 125, in on_shutter
self.handle_worker(worker)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 77, in handle_worker
update_freq=self.worker_update_freq,
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/managers.py", line 62, in update_heartbeat
defaults={'last_heartbeat': heartbeat},
File "/Users/andrewjones/.virtualenvs/dpet-janitor/lib/python3.7/site-packages/django_celery_monitor/managers.py", line 33, in select_for_update_or_create
obj = self.select_for_update().get(**lookup)
TypeError: get() argument after ** must be a mapping, not str
It looks like it's an issue where the django internal method _extract_model_params
is no longer returning (lookup, params)
but only params
Link to the django source code (search for def _extract_model_params
):
https://docs.djangoproject.com/en/2.1/_modules/django/db/models/query/
https://docs.djangoproject.com/en/2.2/_modules/django/db/models/query/
Start with your steps, and there is data in the wokerstate table, but there is no data in taskstate?
When using django_celery_results
I have to set this to django-db
, but with this packege, when I am running a task and I am query-ing for it's state I am getting:
'DisabledBackend' object has no attribute '_get_task_meta_for'
Currently, calling a task with arguments and kwarguments results in something like this:
>>> task.args
[True]
>>> task.kwargs
{'arg1': 'some_string', 'arg2': False}
This is a bit of a pain when parsing this information, in particular to send it to a Javascript frontend, since it's almost JSON but not quite.
Could there be a setting to enable saving this information as properly encoded JSON? e.g.
>>> task.args
[true]
>>> task.kwargs
{"arg1": "some_string", "arg2": false}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.