Code Monkey home page Code Monkey logo

django-uwsgi-taskmanager's Introduction

Django uWSGI Taskmanager

Code style: black

Django application to manage async tasks via admin interface, using uWSGI spooler.

See documentation at http://django-uwsgi-taskmanager.rtfd.io/

Features

  • Start and stop your tasks via admin
  • Schedule tasks
  • Plan tasks as cron items
  • Check or download the generated reports/logs
  • Simply write a standard Django Command class (your app doesn't need to interact with Django uWSGI Taskmanager)
  • Get notifications via Slack or email when a task fails

Installation

  1. Install the app with pip:

    • via PyPI:

      pip install django-uwsgi-taskmanager

    • or via GitHub:

      pip install git+https://github.com/openpolis/django-uwsgi-taskmanager.git

  2. Add "taskmanager" to your INSTALLED_APPS setting like this:

    INSTALLED_APPS = [
        "django.contrib.admin",
        # ...
        "taskmanager",
    ]
  3. Run python manage.py migrate to create the taskmanager tables.

  4. Run python manage.py collectcommands to create taskmanager commands.

  5. Include the taskmanager URLConf in your project urls.py like this (optional):

    from django.contrib import admin
    from django.urls import include, path
    
    urlpatterns = [
        path("admin/", admin.site.urls),
        path("taskmanager/", include("taskmanager.urls")),
    ]
  6. Set parameters in your settings file as below (optional):

    UWSGI_TASKMANAGER_N_LINES_IN_REPORT_INLINE = 10
    UWSGI_TASKMANAGER_N_REPORTS_INLINE = 3
    UWSGI_TASKMANAGER_SHOW_LOGVIEWER_LINK = True
    UWSGI_TASKMANAGER_USE_FILTER_COLLAPSE = True
    UWSGI_TASKMANAGER_SAVE_LOGFILE = False
    

Usage

You just need to install django-uwsgi-taskmanager in your Django Project and run collectcommands as described. Django uWSGI Taskmanager will collect all the commands and make them available for asynchronous scheduling in the admin.

If you need a new asynchronous task, just write a standard custom Django command, and synchronize the app. Then go to the admin page and schedule it.

You can disable some commands from the admin, and let users (with limited permissions) schedule only the available ones.

uWSGI ini file (vassal) has to include the spooler and pythonpath option.

NOTE: remember to manually create the spooler directory with right permissions before start uWSGI

Enabling notifications

To enable Slack notifications support for failing tasks, you have to first install the required packages, which are not included by default. To do that, just:

pip install django-uwsgi-taskmanager[notifications]

This will install the django-uwsgi-taskmanager package from PyPI, including the optional dependencies required to make Slack notifications work.

Email notifications are instead handled using Django django.core.mail module, so no further dependencies are needed and they should work out of the box, given you have at least one email backend properly configured.

Then, you have to configure the following settings:

  • UWSGI_TASKMANAGER_NOTIFICATIONS_SLACK_TOKEN, which must be set with you own Slack token as string.
  • UWSGI_TASKMANAGER_NOTIFICATIONS_SLACK_CHANNELS, a list of strings representing the names or ids of the channels which will receive the notifications.
  • UWSGI_TASKMANAGER_NOTIFICATIONS_EMAIL_FROM, the "from address" you want your outgoing notification emails to use.
  • UWSGI_TASKMANAGER_NOTIFICATIONS_EMAIL_RECIPIENTS, a list of strings representing the recipients of the notifications.

Demo

This a basic Django demo project with a uwsgi.ini file and four directories (media, spooler, static, venv).

demo/
├── demo/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── media/
├── spooler/
├── static/
├── uwsgi.ini
└── venv/

This is the content of uwsgi.ini file required to execute the project with Django:

[uwsgi]
chdir = %d
env = DJANGO_SETTINGS_MODULE=demo.settings
http-socket = :8000
master = true
module = demo.wsgi
plugin = python3
pythonpath = %d
spooler = %dspooler
static-map = /static/=%dstatic
virtualenv = %dvenv

Try the demo project

Enter the demo directory, then create and activate the demo virtual environments:

$ cd demo
$ mkdir -p venv
$ python3 -m venv ./venv
$ source ./venv/bin/activate

Install Django uWSGI taskmanager:

(venv) $ pip install django-uwsgi-taskmanager

Install uWSGI (if you use uWSGI of your OS you can skip this step):

(venv) $ pip install uwsgi

Collect all static files:

(venv) $ python manage.py collectstatic

Create all the tables:

(venv) $ python manage.py migrate

Collect all commands:

(venv) $ python manage.py collectcommands

Create a super user to login to the admin interface:

(venv) $ python manage.py createsuperuser

Start the project with uWSGI:

(venv) $ uwsgi --ini uwsgi.ini

Visit http://127.0.0.1:8000/admin/

Copyright

Django uWSGI taskmanager is an application to manage async tasks via admin interface, using uWSGI spooler.

Copyright (C) 2019-2020 Gabriele Giaccari, Gabriele Lucci, Guglielmo Celata, Paolo Melchiorre

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

django-uwsgi-taskmanager's People

Contributors

gabrielelucci avatar guglielmo avatar pauloxnet avatar unusual-thoughts avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

django-uwsgi-taskmanager's Issues

Tasks with status SPOOLED but missing spooled task files never run

Hi,

I suppose this is kind of an odd one...not necessarily an issue...

So, I've got a Docker service using a Dockerized Django app that uses your taskmanager (which is great, btw).

I start up the service (after collecting all the commands). I create a scheduled task, start it, and then watch it run at the next scheduled time. Everything seems fine.

Then, I bring down the service and back up again. I believe this causes all the spool files in the container to be deleted (I'm not using a volume). The tasks I created are still there with the status SPOOLED, but they never run again at their scheduled times. (Again, I'm guessing because the spool files are no longer there.) In order to get them to run, I have to "Start" each of the tasks again (which I believe creates the spool files).

This is a mild inconvenience. I could create a volume that attaches to the container so the spool files are never removed, but there is also the case where the tasks themselves are in the Django database but (for some reason) that volume is gone (i.e. the spool files are gone).

Is this a known issue/bug/feature/whatever?

Tx,

Ryan

PS my requirements.txt

alabaster==0.7.12
appdirs==1.4.4
asgiref==3.4.1
attrs==21.2.0
autopage==0.4.0
autopep8==1.5.7
Babel==2.9.1
base58==2.1.0
bcrypt==3.2.0
beautifulsoup4==4.9.1
certifi==2021.5.30
cffi==1.14.6
charset-normalizer==2.0.4
cliff==3.9.0
cmd2==2.1.2
colorama==0.4.4
coreapi==2.3.3
coreschema==0.0.4
cryptography==3.4.8
debtcollector==2.3.0
decorator==5.0.9
Django==3.1.6
django-appconf==1.0.4
django-auth-ldap==2.2.0
django-better-admin-arrayfield==1.3.0
django-compressor==2.4
django-debug-toolbar==2.2
django-extensions==3.0.5
django-filter==2.3.0
django-guardian==2.3.0
django-mysql==3.8.1
django-nested-admin==3.3.2
django-netfields==1.2.2
django-sass-processor==1.0.0
django-simple-history==2.11.0
django-sslserver==0.22
django-uwsgi==0.2.2
django-uwsgi-taskmanager==2.2.12
django-widget-tweaks==1.4.8
djangorestframework==3.11.1
djangorestframework-csv==2.1.0
djangorestframework-guardian==0.3.0
djangorestframework-simplejwt==4.8.0
docutils==0.17.1
dogpile.cache==1.1.4
drf-yasg==1.20.0
et-xmlfile==1.1.0
fabric==2.5.0
file-read-backwards==2.0.0
futurist==2.4.0
gnocchiclient==7.0.6
idna==3.2
imagesize==1.2.0
inflection==0.5.1
invoke==1.6.0
iso8601==0.1.16
itypes==1.2.0
jdcal==1.4.1
Jinja2==3.0.1
jmespath==0.10.0
jsonpatch==1.32
jsonpointer==2.1
jsonschema==3.2.0
keystoneauth1==4.3.1
libsass==0.20.0
lxml==4.5.2
Markdown==3.2.2
MarkupSafe==2.0.1
monotonic==1.6
msgpack==1.0.2
munch==2.5.0
mysqlclient==2.0.1
netaddr==0.8.0
netifaces==0.11.0
openpyxl==3.0.6
openstacksdk==0.59.0
os-client-config==2.1.0
os-service-types==1.7.0
osc-lib==2.4.2
oslo.config==8.7.1
oslo.context==3.3.1
oslo.i18n==5.1.0
oslo.log==4.6.0
oslo.serialization==4.2.0
oslo.utils==4.10.0
packaging==21.0
paramiko==2.7.2
pbr==5.6.0
prettytable==0.7.2
psycopg2-binary==2.8.5
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pycparser==2.20
Pygments==2.10.0
pyinotify==0.9.6
PyJWT==2.1.0
PyNaCl==1.4.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyperclip==1.8.2
pyrsistent==0.18.0
python-cinderclient==7.1.0
python-dateutil==2.8.2
python-designateclient==4.1.0
python-glanceclient==3.2.1
python-keystoneclient==4.1.0
python-ldap==3.3.1
python-monkey-business==1.0.0
python-neutronclient==7.2.0
python-novaclient==17.2.0
python-octaviaclient==2.1.0
python-openstackclient==5.3.1
pytz==2021.1
PyYAML==5.3.1
rcssmin==1.0.6
requests==2.26.0
requests-aws==0.1.8
requestsexceptions==1.4.0
rfc3986==1.5.0
rgwadmin==2.3.1
rjsmin==1.1.0
ruamel.yaml==0.17.16
ruamel.yaml.clib==0.2.6
simplejson==3.17.5
six==1.16.0
snowballstemmer==2.1.0
soupsieve==2.2.1
Sphinx==3.2.1
sphinx-rtd-theme==0.5.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sqlparse==0.4.1
stevedore==3.4.0
toml==0.10.2
ujson==4.1.0
unicodecsv==0.14.1
uritemplate==3.0.1
urllib3==1.26.6
uWSGI==2.0.19.1
warlock==1.3.3
wcwidth==0.2.5
wrapt==1.12.1

Notification system at the end of a task

At the end of a task, if the result is not OK, the system sends a notification to interested parties.

The interested parties (email addresses or slack channels), are defined in the dedicated settings.

The notification system(s) are:

  • email, through django (email settings must already have been configured)
  • slack, through an integration within the application
    They can be selected through the dedicated settings. It is possible to select one or both of them.

The dedicated settings, is a section in django's settings and will contain all configuration parameters, with the environment variables where keys, usernames and passwords can be read.
The section must be added when the django-uwsgi-taskmanager is installed.

The notification messages should be generated automatically, by any task, without the necessity to write any code within the task.

Notifications can be disabled for a single task, using an argument passed in its invocation.

The notification message can be generated within the task, by overriding a method. This will substitute the automatically generated message.

The automatically generated message should contain:

  • the name of the task and the parameters
  • the time of start and time of end of the execution of the task
  • the number of warnings and/or errors generated in the log
  • the failure message (if any)
  • a link to the log report

In case everything is OK, and no errors, nor warnings are generated in the logs, no notification is sent.

Next ride must start at the given time, not at the end of the task

Currently periodic tasks next ride timestamp is computed by adding the deriodicity (6 hours, 1 day, 1 week) at the current time, whenever a task ends.

This means that if a task is scheduled at 3.am, with periodicity of 1 day, and takes 20 minutes, it will start at least 20 minutes later every day. This could impact daily operations.

It is important that tasks start, at least tentatively, at the indicated time, so that their impact on given operations are minimised.

Error user Windows

I'm a windows user and the stacktrace says it's going to use os.uname, but for windows it has to be platform.uname()...tips?


× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\luxu\AppData\Local\Temp\pip-install-jnxljlnv\uwsgi_63581899bbb249248ac16d1f216e6cce\setup.py", line 3, in
import uwsgiconfig as uc
File "C:\Users\luxu\AppData\Local\Temp\pip-install-jnxljlnv\uwsgi_63581899bbb249248ac16d1f216e6cce\uwsgiconfig.py", line 8, in
uwsgi_os = os.uname()[0]
AttributeError: module 'os' has no attribute 'uname'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

Better explanation of the goal and features

Feedback form @bittner

  • do a short CLI demo video with asciinema (or similar) to showcase what it does
  • explain if it's meant as a replacement to Celery, Flower, Jobtasic or something completely different
  • explain in the README what it is not

Changing scheduling and repetition fields leaves unwanted files in the spooler

Whenever the scheduling fields of an already scheduled task are changed,
the old scheduled execution, re-creates a new file in the spooler with the old scheduling values.

Thus, multiple files, with different execution scheduling will live in the spooler, generating confusion and unwanted executions.

A workaround is to manually stop the task (thus removing the file in the spooler), before changing the values.

This could be also the possible solution. Whenever an update is required for a task, the task is automatically stopped before the update, so that no trailing files are left in the spooler.

Respawning uwsgi processes causes multiple task invocations

Whenever a uwsgi process dies, in the middle of a spooled task execution, it is respawned.

Since this happens before the task is marked as processed, and removed from the spooler (or re-spooled with a later execution time), the respawned uwsgi process proceeds with reading the file in the spooler once again, untill the crash causes are resolved.

This can potentially cause hundreds of invocations per minutes, flooding the disk with log files.

A mechanism to control the re-spawning should be used.

DatabaseError on report save

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uwsgidecorators.py", line 64, in manage_spool_request
    ret = f(*vars['args'], **vars['kwargs'])
  File "/usr/local/lib/python3.9/site-packages/taskmanager/tasks.py", line 107, in exec_command_task
    report_obj.save(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in _save_table
    raise DatabaseError("Save with update_fields did not affect any rows.")
django.db.utils.DatabaseError: Save with update_fields did not affect any rows.

Real-time log streaming for the report

During the execution of a task it is currently possible to follow the messages produced in the report only by constantly refreshing the report page.

An enhancement is a live report page that works with automatic periodic ajax requests, refreshing the page content every 5 or 10 seconds. Each request only get the new log messages, differentially.

The live report page link or button is only available during a task's execution (status is started).

The page is shown when the button or link is pressed, as a full html page, with javascript that enables automatic reload of new messages, and buttons to:

  • go to the top of the log messages
  • go to the bottom of the log messages (enter the follow log stream mode, too)
  • show all raw content
    By default, and whenever the follow log stream mode is on, new messages are added and the content is scrolled down to the bottom, so that following the log messages is easy.

The page allows search and filtering capabilities.

  • a text can be searched within the content (similar to grep linux command): only lines containing the text will be shown; the searched text will be highlighted;
  • messages of one or more levels can be selected (debug, info, warning, error); one or more levels can be chosen.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.