Comments (96)
@mnaberez I propose that issue be reopened, because the proposed fix in #477 is really not correct at all for the use cases that are requested here. I will try to recap and explain what I mean:
The idea is that a common alias with the name graceful
or reload
or something like that should be configurable to mean "send arbitrary signal foo to the process under supervision". The reason is that different software uses different signals for these actions, typically one of HUP or USR1 or USR2, and with different specific meanings assigned to each signal. By allowing third party code (e.g. sysadmin scripts to manage config files) to specify the signal they want to send, you leak implementation details to the third party script. The third party does does not care which signal to send it just wants to say to supervisor _"Hey, I changed this application config file, so do whatever you need to tell this process to reload its configs."_
I strongly suggest that a more useful config and command for this use case would be something like:
[program:foo]
stopsignal=TERM
restartsignal=HUP
reloadsignal=USR2
supervisor restart foo # this is a hard restart
supervisor reload foo # this is a graceful reload
Again, different tools implement their signals differently, and the idea is that supervisor should abstract these differences to a reasonable degree. For example, in nginx, graceful restart is SIGHUP: http://nginx.org/en/docs/control.html But in Apache, hard restart is SIGHUP and graceful restart is SIGUSR2: https://httpd.apache.org/docs/current/stopping.html Meanwhile, in PHP-FPM, there is no hard restart (you would SIGTERM and begin again) but graceful restart is also SIGUSR2.
What's really important is to provide a way to abstract these command-to-signal mappings and allow sysadmins to configure the application's supervisor configs once, and then reuse the commands elsewhere without having to remember that nginx needs a HUP but Apache needs a USR2 all over their other automation scripts.
from supervisor.
Hi, as I mentioned in a comment in another thread, I think the ability to signal supervisor processes is great, but I think there's still a good argument to provide a "graceful restart" command. Here's a common use case:
- An application which consists of a PHP app running on Apache, a flask/gunicorn web app and a few celery worker processes.
- I have grouped these into a supervisor process group. A configuration has changed and I want to restart them all.
- In this case I have to send
SIGUSR1
to Apache,SIGHUP
to the gunicorn process, andSIGTERM
to the celery workers.
It would be nice to be able to issue supervisorctl graceful myapp:*
and supervisor knows which signal to send in each instance, while defaulting to a normal "restart" if not available. This could be done by allowing an optional "gracefulsignal" to be included in the program configuration:
[program:gunicorn]
command=gunicorn wsgi:app -c conf/gunicorn.py
directory=/myapp
autostart=true
autorestart=true
gracefulsignal=HUP
from supervisor.
I added frontend (supervisorctl) support to #477 just now. Looks like this:
supervisor> help signal
signal <signal name> <name> Signal a process
signal <signal name> <gname>:* Signal all processes in a group
signal <signal name> <name> <name> Signal multiple processes or groups
supervisor> signal HUP dog:3 dog:4
dog:3: signalled
dog:4: signalled
supervisor> signal HUP dog:*
dog:1: signalled
dog:0: signalled
dog:3: signalled
dog:2: signalled
dog:4: signalled
supervisor> signal USR1 dog:1 dog:2
dog:1: signalled
dog:2: signalled
from supervisor.
I am using supervisorctl pid appname | xargs kill -s HUP
for gunicorn, but a proper reload method, instead of relying on hackings, would be very welcome.
from supervisor.
Oh, I get what you're saying. But guincorn supports a graceful shut down with the QUIT signal. So I use stopsignal=QUIT
in my conf file for reloads without cutting off a process.
from supervisor.
If your program foo
supports graceful restart by receiving signal SIGUSR2
, then you would use the command supervisorctl signal sigusr2 foo
instead of the command supervisorctl restart foo
.
from supervisor.
@mnaberez I propose that issue be reopened too, and i do not think supervisorctl signal sigusr2 foo
useful for graceful restart.
The process foo(pid: 60900) for example under supervisor control, we send sigusr2 signal to it for graceful restart, the process(pid: 60900) receive the signal sigusr2, it will fork or exec a child process(pid: 60901) with some extra parameters such as the the active listeners, then this child process will close the parent process(pid:60900) by send SIGTERM to it, then the parent process(pid: 60900) exit. BUT NOW SUPERVISIOR WILL RESTART THE PROCESS TOO, the new process(pid: 60902) started by SUPERVISIOR may start failed due to the net port address already in use by process 60901. But 60901 will not under supervisor control.
from supervisor.
1,000 times this
from supervisor.
+1
I'd really like to be able to use graceful restarts for uwsgi, celeryd, etc. that I have running under supervisord.
from supervisor.
Big +1.
Also note that you can't use the supervisorctl pid [processname] | awk | xargs kill -SIGNAL
hack for groups that spawn more than one process.
from supervisor.
Another +1 for this!
from supervisor.
Another +1
from supervisor.
This would be great, as I could perform deploys under gunicorn without bringing my site down.
from supervisor.
Isn't this already handled with stopsignal
? I use it for gunicorn all the time.
from supervisor.
@mattrobenolt I don't think so, since then running supervisorctl stop gunicorn
would not actually stop gunicorn. It would just reload it.
from supervisor.
Stopsignal doesn't solve our problem. Supervisor still expects the process
to stop and will eventually sigkill it.
Having a restartsignal option would be nice...
from supervisor.
If there's interest from the maintainer I will write a patch for this...
from supervisor.
definitely +1 on this!
from supervisor.
+1
from supervisor.
any word on this?
from supervisor.
+1 this would be very useful, in fact that's the only thing important feature I'm missing from Supervisor which is otherwise greate piece of software
from supervisor.
+2
from supervisor.
Also +1 from me! (Without this feature I cannot watch my nginx with supervisor, and I would love to, because supervisor is just so good!)
Anyone from the supervisor team reading this? What is the status on this feature request?
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
π
from supervisor.
π
from supervisor.
π
from supervisor.
haha me too. It would be the best thing the ability to describe custom commands in config per program
from supervisor.
π
from supervisor.
Any update on this issue ? There seems to be an unreviewed patch pending.
from supervisor.
+1
from supervisor.
Related: #179
from supervisor.
+1
from supervisor.
Another +1 for this, as I'd love to see unicorn zero-downtime deploys working nicely in supervisord
However, I believe the problem is bigger than simply having supervisord support a configuration option for a restart signal. For a thorough discussion see ( http://blog.nicolai86.eu/posts/2012-11-28/zero-downtime-deployments-with-unicorn-and-supervisord/ ) and the mentioned tool ( https://github.com/alphagov/unicornherder/blob/master/unicornherder/herder.py )
Long story short: Many of these zero-downtime flows involve the master (that supervisord is managing) renaming itself to master-old, forking a new master-new, and when all the init and children are ready, old ones commit suicide and new ones accept requests.
While the signalling mechanism (typically sigUSR2) is, I believe, not challenging to add in supervisord, the bigger issue is master-new which supervisord 1. Knows little about and 2. Is not its parent
I feel that for this to become turn-key in supervisord, without the addition of even more layers like the unicornherder glue, supervisord would need to be able to "watch" the given PID file and be able to learn that a given [program]'s PID has changed and start watching the new PID. I'm unsure whether the fact that master-new isn't a child of supervisord is a big deal or not.
from supervisor.
Right, but not ALL of the zero-downtime workflows require that. The most popular one, just using HUP to gracefully reload the gunicorn config and restart all the workers when they finish processing their current requests, would be great to have as a supervisord capability.
from supervisor.
π
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1 for simple HUP (gunicorn reload usecase)
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
It would be really useful if supervisor will allow sending custom signals via supervisorctl and web control pannel
from supervisor.
People, this is not 'Nam. This is open source. There are rules.
Quit feature requesting and start coding it your damn selves
from supervisor.
To send custom signals, you can use the mr.laforge plugin.
from supervisor.
This is open source. There are rules.
Quit feature requesting and start coding it your damn selves
Of course. However, I've always seen this as letting current regular contributors know which suggestions / enhancements are most wanted by their community. Furthermore there's a pull req open which seems to implement this (though I assume it'll need a minor update since it was made 11 months ago). Maybe the pull req was unsatisfactory, but there's no comment on it to indicate that.
The mr.laforge plugin seems nice, if it works on a few older versions of supervisor besides newest stable, it'll probably do.
from supervisor.
+1
from supervisor.
+1
from supervisor.
π
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
π
from supervisor.
+1 !
from supervisor.
+1
from supervisor.
+1
from supervisor.
I took PR #228 and updated it to create a PR based on the latest master: #477. I hope that's useful.
from supervisor.
As @fschulze already mentioned, the mr.laforge package supplies a supervisor plugin that can be used to send signals to processes:
supervisorctl kill HUP nginx
That said, it would be nice to have this built into supervisor, since it's such a common use case and tons of people want it. One could examine what mr.laforge does and use it as inspiration for creating a PR to add built-in support to supervisor.
from supervisor.
+1 on @adnam 's suggestion
from supervisor.
+1 on this request, irrespective of implementation details. I am interested in using the USR2 signal facility for nginx ( http://nginx.org/en/docs/control.html#upgrade ) (yes, we run nginx under supervisord !) to do on the fly nginx upgrades. Unfortunately this currently doesn't work since sending a USR2 signal to nginx (ie: outside of supervisord context) creates a new nginx master process which is not a child of supervisord.
Update: just saw pull #477 ...that'll work, methinks
from supervisor.
+1 Hope to reload gracefully
from supervisor.
Finally I set the stopwaitsecs
long enough to work around my problem.
The number of seconds to wait for the OS to return a SIGCHILD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHILD from the process, supervisord will attempt to kill it with a final SIGKILL.
from supervisor.
apt-get install supervisor
My version is:
$ dpkg -l |grep -i supervisor
ii supervisor 3.0b2-1
restart the program
$ supervisorctl restart program-name
i.e.
$ supervisorctl restart order-form
order-form: stopped
order-form: started
from supervisor.
+1 for graceful reload, where we can actually choose the graceful signal to send :)
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
There are 390 forks for this. Does anyone know if there is one that implements this change?
Edit: https://github.com/fedosov/supervisor appears to have the change, though I haven't tested it for myself yet.
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1
from supervisor.
+1 ... we should have a party at nov 22.
from supervisor.
+1. I'll bring champagne
from supervisor.
+1 for supporting signal and graceful shutdown/restart.
from supervisor.
My first impression was that when supervisord receive some signal to act as a proxy and send the same signal to the controlling processes. But the signals sent to supervisord are not related on how supervisord communicate with the controlled processes.
from supervisor.
Since Supervisor 3.2 you can send any signal to a subprocess as shown above.
from supervisor.
Thanks, I saw it, I have also read all the comments under the pull request #477 and I am curious to see how supervisor will choose (or not) to support rolling/graceful restarts.
from supervisor.
The signal
implementation in #477 looks great β I think it should suffice for what most folks on this thread want. Looking forward to finally using it π
from supervisor.
@zsounder the problem you are describing is not one that supervisor can solve right now - if your process does a fork and exec then it will escape from under supervisor, and that limitation is different and unrelated to this issue of abstracting the restart signals. Again the choice of restart signals is program specific and is based on the behavior of the program that the administrator is writing their supervisor configs for. You need to do your own reading on which signals to use.
My very specific argument here is that once you the administrator figure out which signal you want to send, supervisor should help abstract the decision rather than help leak the decision into other parts of your system.
For your use case with fork exec, you might look into cgroups on Linux, because it allows you to manage a subtree of processes together.
from supervisor.
@sodabrew perhaps you are right, in fact, i think supervisor should not just monitor the process by process id, monitor by the process name could be helpful tooγ
def monitor_process(key_word, cmd):
if check_process_by_name(key_word):
return
sys.stderr.write('process[%s] is lost, run [%s]\n' % (key_word, cmd))
subprocess.call(cmd, shell=True)
a simple implementation for this, we could run it under crontab.
If supervisor support Monitor The Process By Process Name, Even The Port Process Listened on, it will be useful i think.
from supervisor.
@zsounder Supervisor does not monitor by pid, per se, it monitors by parent-child relationship. Please read http://supervisord.org/subprocess.html to learn more about its design. Please please please do not hijack this thread to propose new ideas on UNIX process management.
from supervisor.
@sodabrew I see now, ths
from supervisor.
mark
from supervisor.
mark
from supervisor.
Related Issues (20)
- "Unable to Connect Processes" HOT 1
- 4.2.1 Update username and password not work HOT 1
- Supervisor doesn't work after installing from codebase (version 4.3) HOT 1
- error: <class 'xml.parsers.expat.ExpatError'>, unclosed token: line 130, column 0: file: /usr/lib/python3.10/xmlrpc/client.py line: 460
- It is possible to make supervisor logs to be in JSON format?
- Is it possible to add a new choice for placing the config file? HOT 1
- crashmail: missing for EXITED HOT 1
- HTTP/1.1 500 Internal Server Error returned for "http://127.0.0.1:9551/RPC2". HOT 9
- gunicorn worker sometimes got stuck when manage by supervisor
- Export supervisorctl status running process to azure application insight HOT 1
- Locale & Date python with Supervisord
- New release on Pypi HOT 3
- Fatal error after update python from 3.10.13 to 3.10.14 - ModuleNotFoundError: No module named 'xmlrpclib' HOT 6
- Unwanted restart when modifying config HOT 1
- How to pass custom variables HOT 1
- Feature Request: Option to Disable HTTP Server Startup HOT 1
- Random Backoff for Load Distribution
- Supervisor 4.2.5 not setting env vars HOT 1
- Variables defined in environment= in [program:x] section cannot be reused in %(ENV_X)s expansions in the same section
- Create /var/log/supervisor/ if not exists HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from supervisor.