Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compatibility of systemd with zero downtime reloads with SIGUSR2 #2856

Closed
natefoo opened this issue Aug 25, 2022 · 8 comments · May be fixed by #3285
Closed

Compatibility of systemd with zero downtime reloads with SIGUSR2 #2856

natefoo opened this issue Aug 25, 2022 · 8 comments · May be fixed by #3285

Comments

@natefoo
Copy link

natefoo commented Aug 25, 2022

Systemd expects the process it execs to continue running and stay foregrounded, which is incompatible with the way that SIGUSR2 zero-downtime reloading works. This is because after sending the final signal in the series of USR2 -> WINCH -> TERM, the old master exits, leaving the new master running as a new (and also detached) pid. At this point, because the foreground process has exited, systemd terminates everything in the process group (or cgroup) and you're left with nothing.

I had assumed that it was expected that these two things were incompatible (i.e. this is exactly why unicornherder exists), but in #1109 it seems to be implied that it would in fact work. I am not sure how this could be possible from a conceptual standpoint, though, and I have indeed been unable to make it work.

I suspect this issue is not enormous for many others because they have alternative solutions (unicornherder, kubernetes), or their application startup time is short enough that a HUP reload isn't a big deal. However, our startup time can be minutes long due to factors that can't easily be addressed, so zero downtime restarts are pretty important.

We aren't opposed to using unicornherder if this is the best solution, but it's unclear whether it's receiving active maintenance (we've had some minor PRs open on it for a while) and so wanted to explore other solutions. It seems like the SIGUSR2 process is not expected to work under systemd, but it would be great to get some confirmation of that, since the docs don't say, and we'd love to know if there are other solutions that are recommended for this.
@benoitc
Copy link
Owner

benoitc commented Sep 7, 2022 •
edited
Loading

With Sytemd you probably need to set ExecReload in your service file. Something like this:

ExecReload=/path/to/gunicorn --check-config
ExecReload=/bin/kill -HUP $MAINPID

Doing that way you can use the command line systemctl reload gunicorn.service which should do what you want when using tooling like systemd and doesn't start gunicorn in daemon mode. Or replace HUP by the USR2 signal. Graceful reload normally take care about long restart if you're behind a proxy with enough timeout.

@natefoo
Copy link
Author

natefoo commented Sep 7, 2022

I tried this with USR2 but couldn't get past this error on EL (Rocky) 8:

[2022-09-07 13:48:03 +0000] [654099] [INFO] Starting gunicorn 20.1.0
Traceback (most recent call last):
  File "/home/galaxy/galaxy/.venv/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 198, in run
    self.start()
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 155, in start
    self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)
  File "/home/galaxy/galaxy/.venv/lib/python3.9/site-packages/gunicorn/sock.py", line 170, in create_sockets
    sock = socket.fromfd(fd, socket.AF_UNIX, socket.SOCK_STREAM) 
  File "/home/galaxy/conda/envs/[email protected]/lib/python3.9/socket.py", line 545, in fromfd
    return socket(family, type, proto, nfd)
  File "/home/galaxy/conda/envs/[email protected]/lib/python3.9/socket.py", line 232, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 88] Socket operation on non-socket

It was encountered by someone else in #1898 but they fixed it by reinstalling the OS - I am deploying this on a fresh Rocky 8 install on a VM created specifically to test this. On Debian 11 I get the above error or this:

[2022-09-07 10:40:22 -0400] [4038266] [INFO] Starting gunicorn 20.1.0 
Traceback (most recent call last):
  File "/home/nate/work/galaxy2/.venv/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()  
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 198, in run
    self.start()
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 155, in start
    self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/sock.py", line 173, in create_sockets
    listener = sock_type(sock_name, conf, log, fd=fd)
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/sock.py", line 117, in __init__
    super().__init__(addr, conf, log, fd=fd)
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/sock.py", line 31, in __init__
    self.sock = self.set_options(sock, bound=bound)
  File "/home/nate/work/galaxy2/.venv/lib/python3.9/site-packages/gunicorn/sock.py", line 56, in set_options
    sock.listen(self.conf.backlog)
OSError: [Errno 22] Invalid argument

I am still confused how USR2 reloading would work even if I was able to get past this error, because systemd would still kill the new processes after the old master is killed with KILL, wouldn't it?

@natefoo natefoo mentioned this issue Sep 29, 2022
Closed
@benoitc
Copy link
Owner

benoitc commented Oct 14, 2022

did you try this setup ? It seems you let gunicorn starting its own socket there. Systemd like to work its own way.

@natefoo
Copy link
Author

natefoo commented Oct 17, 2022

The above examples were using the systemd socket activation method given in the gunicorn docs, so it should not have been creating its own socket.

@benoitc
Copy link
Owner

benoitc commented Oct 18, 2022

please show me your configuration.

@natefoo
Copy link
Author

natefoo commented Oct 19, 2022

Sure, here is the simplest example I can demonstrate with:

deb:~$ . ~/.virtualenvs/gunicorn-test/bin/activate
(gunicorn-test) deb:~$ pip list
Package    Version
---------- -------
gunicorn   20.1.0
pip        22.3
setuptools 65.3.0
wheel      0.37.1
deb:~$ cat ~/gunicorn-test/hello.py 
def app(environ, start_response):
    data = b"Hello, World!\n"
    start_response("200 OK", [
        ("Content-Type", "text/plain"),
        ("Content-Length", str(len(data)))
    ])
    return iter([data])
deb:~/gunicorn-test$ cat /etc/systemd/system/gunicorn.socket 
[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock
# Our service won't need permissions for the socket, since it
# inherits the file descriptor by socket activation
# only the nginx daemon will need access to the socket
SocketUser=www-data
# Optionally restrict the socket permissions even more.
# SocketMode=600

[Install]
WantedBy=sockets.target
deb:~$ cat /etc/systemd/system/gunicorn.service 
[Unit]
Description=gunicorn
Requires=gunicorn.socket
After=network.target

[Service]
Type=notify
# the specific user that our service will run as
User=nate
Group=nate
# another option for an even more restricted service is
# DynamicUser=yes
# see http://0pointer.net/blog/dynamic-users-with-systemd.html
RuntimeDirectory=gunicorn
WorkingDirectory=/home/nate/gunicorn-test
ExecStart=/home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
ExecReload=/bin/kill -s USR2 $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Running the service:

deb:/# systemctl daemon-reload
deb:/# systemctl enable --now gunicorn.socket
Created symlink /etc/systemd/system/sockets.target.wants/gunicorn.socket → /etc/systemd/system/gunicorn.socket.
deb:/# curl --unix-socket /run/gunicorn.sock http
Hello, World!
deb:/# systemctl status gunicorn                 
● gunicorn.service - gunicorn
     Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; preset: enabled)
     Active: active (running) since Tue 2022-10-18 23:45:08 EDT; 2s ago
TriggeredBy: ● gunicorn.socket
   Main PID: 1450789 (gunicorn)
     Status: "Gunicorn arbiter booted"
      Tasks: 5 (limit: 38331)
     Memory: 30.3M
        CPU: 134ms
     CGroup: /system.slice/gunicorn.service
             ├─1450789 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450820 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450834 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450836 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             └─1450837 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app

Oct 18 23:45:07 deb systemd[1]: Starting gunicorn...
Oct 18 23:45:08 deb gunicorn[1450789]: [2022-10-18 23:45:08 -0400] [1450789] [INFO] Starting gunicorn 20.1.0
Oct 18 23:45:08 deb gunicorn[1450789]: [2022-10-18 23:45:08 -0400] [1450789] [INFO] Listening at: unix:/run/gunicorn.sock (1450789)
Oct 18 23:45:08 deb gunicorn[1450789]: [2022-10-18 23:45:08 -0400] [1450789] [INFO] Using worker: sync 
Oct 18 23:45:08 deb systemd[1]: Started gunicorn.
Oct 18 23:45:08 deb gunicorn[1450820]: [2022-10-18 23:45:08 -0400] [1450820] [INFO] Booting worker with pid: 1450820
Oct 18 23:45:08 deb gunicorn[1450834]: [2022-10-18 23:45:08 -0400] [1450834] [INFO] Booting worker with pid: 1450834
Oct 18 23:45:08 deb gunicorn[1450836]: [2022-10-18 23:45:08 -0400] [1450836] [INFO] Booting worker with pid: 1450836
Oct 18 23:45:08 deb gunicorn[1450837]: [2022-10-18 23:45:08 -0400] [1450837] [INFO] Booting worker with pid: 1450837

Attempt a reload:

deb:/# systemctl reload gunicorn
deb:/# systemctl status gunicorn
● gunicorn.service - gunicorn
     Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; preset: enabled)
     Active: active (running) since Tue 2022-10-18 23:45:08 EDT; 5min ago
TriggeredBy: ● gunicorn.socket
    Process: 1454263 ExecReload=/bin/kill -s USR2 $MAINPID (code=exited, status=0/SUCCESS)
   Main PID: 1450789 (gunicorn)
     Status: "Gunicorn arbiter booted"
      Tasks: 5 (limit: 38331)
     Memory: 30.4M
        CPU: 352ms
     CGroup: /system.slice/gunicorn.service
             ├─1450789 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450820 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450834 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             ├─1450836 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app
             └─1450837 /home/nate/.virtualenvs/gunicorn-test/bin/python /home/nate/.virtualenvs/gunicorn-test/bin/gunicorn -w 4 hello:app

Oct 18 23:50:40 deb systemd[1]: Reloading gunicorn...
Oct 18 23:50:40 deb systemd[1]: Reloaded gunicorn. 
Oct 18 23:50:40 deb gunicorn[1450789]: [2022-10-18 23:50:40 -0400] [1450789] [INFO] Handling signal: usr2
Oct 18 23:50:40 deb gunicorn[1454264]: [2022-10-18 23:50:40 -0400] [1454264] [INFO] Starting gunicorn 20.1.0
Oct 18 23:50:40 deb gunicorn[1454264]: Traceback (most recent call last): 
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/bin/gunicorn", line 8, in <module>
Oct 18 23:50:40 deb gunicorn[1454264]:     sys.exit(run())    
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
Oct 18 23:50:40 deb gunicorn[1454264]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
Oct 18 23:50:40 deb gunicorn[1454264]:     super().run()
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
Oct 18 23:50:40 deb gunicorn[1454264]:     Arbiter(self).run()
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/arbiter.py", line 198, in run
Oct 18 23:50:40 deb gunicorn[1454264]:     self.start()
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/arbiter.py", line 155, in start 
Oct 18 23:50:40 deb gunicorn[1454264]:     self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/home/nate/.virtualenvs/gunicorn-test/lib/python3.9/site-packages/gunicorn/sock.py", line 170, in create_sockets
Oct 18 23:50:40 deb gunicorn[1454264]:     sock = socket.fromfd(fd, socket.AF_UNIX, socket.SOCK_STREAM)
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/usr/lib/python3.9/socket.py", line 545, in fromfd
Oct 18 23:50:40 deb gunicorn[1454264]:     return socket(family, type, proto, nfd)
Oct 18 23:50:40 deb gunicorn[1454264]:   File "/usr/lib/python3.9/socket.py", line 232, in __init__
Oct 18 23:50:40 deb gunicorn[1454264]:     _socket.socket.__init__(self, family, type, proto, fileno)
Oct 18 23:50:40 deb gunicorn[1454264]: OSError: [Errno 88] Socket operation on non-socket

I can set the log-level to debug and send that output if it'd help.

@benoitc
Copy link
Owner

benoitc commented Oct 19, 2022 via email

@benoitc
Copy link
Owner

benoitc commented Aug 6, 2024

no activity since awhile. closing feel free to create a new ticket if needed.

@benoitc benoitc closed this as completed Aug 6, 2024
@pajod pajod mentioned this issue Nov 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Retain argv + fds on re-exec, fix USR2 under systemd by notifying new PID pajod/gunicorn
2 participants
@benoitc @natefoo