pplonski / simple-tasks Goto Github PK

View Code? Open in Web Editor NEW

135.0 9.0 34.0 430 KB

Simple tasks real-time updates with django channels, DRF, celery

License: MIT License

Python 38.91% Shell 4.44% Dockerfile 1.97% HTML 2.34% JavaScript 51.32% CSS 1.02%

django djangorestframework django-channels drf celery websocket redis react-redux rabbitmq

simple-tasks's Introduction

Simple-tasks

Simple tasks real-time updates with django channels, DRF, celery

This repo is showing example how you can use celery tasks and django channels to send tasks progress over the websockets. The client side is written in react and redux.

The task that is executed in celery it is simple add. It accepts two arguments in JSON format, example {"arg1": 2, "arg2": 2}. It will execute as long as the value of the arg1. If the value of arg1 is negative the worker crash will be simulated in the code (crash with segfault). If the value of arg2 is negative then the worker will crash during importing fake package. (Both crash simulations were used to show how we can set FAILURE state even if worker crashed).

In this example the task can be in following states: CREATED, PROGRESS, SUCCESS, FAILURE. The progress value is not stored in database, it is only broadcasted by websockets during task exceution.

The task states and progress are set by:

backend/worker/redis_listener.py which is pubsub on redis. It is updating the progress value.
backend/worker/worker_listener.py which is monitoring celery events. It is setting state values, for example in case of task failure.

Nginx configuration

In this example we are running gunicorn and daphne servers and redirecting requests in nginx:

HTTP is going to gunicorn
websocket is going to daphne

This can be changed, and both HTTP and websocket connectionscan be handled by daphne server.

Running

To run the app you can use docker-compose:

docker-compose up --build -d

To stop the app:

docker-compose down

The app will be accessed at 0.0.0.0:8000.

simple-tasks's People

Contributors

Stargazers

Watchers

simple-tasks's Issues

DNS issue

Technically not an issue with the project, though I discovered a bug related to bad DNS using this package. In my case I use(d) Verizon Fios DNS servers, which decides to send marketing domains for unresolved lookups sending me to unallocated.barefruit.co.uk for the wsgi container instead of via it's docker address. Switching to Open DNS fixed my problem. Not sure where it was trying to do a different resolver lookup - probably a docker hostname lockup issue. Will try and do some more research though if someone else gets the 504 issue from the client on the web socket connection, check your DNS settings

ubuntu:xenial is deprecated

this is the image you use for the django app

add client code

client code is missing

Add task logs to websocket logs?

Hi,

I was wondering if its possible to logs tasks through websockets? for example something like this:
https://youtu.be/R_yWvXWa13U?t=66

i did this using django-tasks which fork from django-rq but the problem is i'm facing issue of high cpu usage. After few minutes cpu is on fire such as 200-400% and i'm unable to reproduce issue. That more seems issue of django channels. I'm sure i'm not logging messages to client in right way due to lack of knowledge. That's some work around.

Organizations

Try organizations

postgres

Hi, thank you for your great repository. I want to have access to my database locally but with the current docker-compose.yml file, line 25 (- ./docker/postgres/data:/var/lib/postgresql) data from postgres are not linked to my local directory in docker/postgres/data.

I changed the line 25 to
- ./docker/postgres/data:/var/lib/postgresql/data

still, there is not any saved data available in my local directory. Do you know any solution for that?

websocket

Hi, I tested the repository and exactly followed the instruction, it run without an error but at the footer of the page (http://127.0.0.1:8000/) websocket status never changed to connected and its always swichting between connecting and connected.

Here is the docker-compose logs wsgiserver:

wsgiserver_1      | ls: cannot access '/app/backend/server/static/client/static/js': No such file or directory
wsgiserver_1      | /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in 
order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
wsgiserver_1      |   """)
wsgiserver_1      | Operations to perform:
wsgiserver_1      |   Apply all migrations: admin, auth, contenttypes, sessions, 
sites, tasks
wsgiserver_1      | Running migrations:
wsgiserver_1      |   No migrations to apply.
wsgiserver_1      |
wsgiserver_1      | 0 static files copied to '/app/backend/server/django_static', 160 unmodified.
wsgiserver_1      | /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in 
order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
wsgiserver_1      |   """)
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [15] [INFO] Starting gunicorn 19.9.0
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [15] [INFO] Listening at: http://0.0.0.0:8000 (15)
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [15] [INFO] Using worker: threadswsgiserver_1      | [2021-07-01 07:40:24 +0000] [18] [INFO] Booting worker with pid: 18
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [19] [INFO] Booting worker with pid: 19
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [21] [INFO] Booting worker with pid: 21
wsgiserver_1      | [2021-07-01 07:40:24 +0000] [23] [INFO] Booting worker with pid: 23
wsgiserver_1      | /usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in 
order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
wsgiserver_1      |   """)

Great repo. One question

I've just pulled this repo. I'm trying to get my own project running with a similar setup so when I run docker compose on and ec2 instance with daphne and gunicorn.
I was just wondering about static files. Does nginx serve static files from django?

I build the repo and got this error Traceback (most recent call last):
File "./manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/dist-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.6/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle
collected = self.collect()
File "/usr/local/lib/python3.6/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 105, in collect
for path, storage in finder.list(self.ignore_patterns):
File "/usr/local/lib/python3.6/dist-packages/django/contrib/staticfiles/finders.py", line 125, in list
for path in utils.get_files(storage, ignore_patterns):
File "/usr/local/lib/python3.6/dist-packages/django/contrib/staticfiles/utils.py", line 23, in get_files
directories, files = storage.listdir(location)
File "/usr/local/lib/python3.6/dist-packages/django/core/files/storage.py", line 313, in listdir
for entry in os.listdir(path):
FileNotFoundError: [Errno 2] No such file or directory: '/app/backend/server/staticfiles'

Please provide steps to for non-docker way of running this app

Can you please provide steps to run this application without docker.

I am able to start django project with only runserver command.

But rest the

Worker initialization
Configuration of daphane, gunicorn with nginx
Client application initialization
this other steps are also need to be clarified.