ServerEngine is a framework to implement robust multiprocess servers like Unicorn.

Main features:

                  Heartbeat via pipe
                      & auto-restart
                 /                \               ---+
+------------+  /   +----------+   \  +--------+     |
| Supervisor |------|  Server  |------| Worker |     |
+------------+      +----------+\     +--------+     | Multi-process
                        /         \                  | or multi-thread
                       /            \ +--------+     |
      Dynamic reconfiguration         | Worker |     |
     and live restart support         +--------+     |
                                                  ---+

Other features:

• logging and log rotation
• signal handlers
• stacktrace and heap dump on signal
• chuser, chgroup and chumask
• changing process names

What you need to implement at least is a worker module which has run and stop methods.

require 'serverengine'

module MyWorker
  def run
    until @stop
      logger.info "Awesome work!"
      sleep 1
    end
  end

  def stop
    @stop = true
  end
end

se = ServerEngine.create(nil, MyWorker, {
  :daemonize => true,
  :log => 'myserver.log',
  :pid_path => 'myserver.pid',
})
se.run

Send TERM signal to kill the daemon. See also Signals section bellow.

Simply set process or thread to worker_type parameter and number of workers to workers parameter.

se = ServerEngine.create(nil, MyWorker, {
  :daemonize => true,
  :log => 'myserver.log',
  :pid_path => 'myserver.pid',
  :workers => 4,
  :worker_type => 'process',
})
se.run

See also Worker types section bellow.

One of typical implementation styles of TCP servers is that a parent process listens socket and child processes accept connections from clients.

You can optionally implement a server module to control the parent process.

module MyServer
  def before_run
    @sock = TCPServer.new(config[:bind], config[:port])
  end

  attr_reader :sock
end

module MyWorker
  def run
    until @stop
      # you should use Cool.io or EventMachine actually
      c = server.sock.accept
      c.write "Awesome work!"
      c.close
    end
  end

  def stop
    @stop = true
  end
end

se = ServerEngine.create(MyServer, MyWorker, {
  :daemonize => true,
  :log => 'myserver.log',
  :pid_path => 'myserver.pid',
  :workers => 4,
  :worker_type => 'process',
  :bind => '0.0.0.0',
  :port => 9071,
})
se.run

ServerEngine logger rotates logs by 1MB and keeps 5 generations by default.

se = ServerEngine.create(MyServer, MyWorker, {
  :log => 'myserver.log',
  :log_level => 'debug',
  :log_rotate_age => 5,
  :log_rotate_size => 1*1024*1024,
})
se.run

See also Configuration section bellow.

Server programs running 24x7 hours need to survive even if a process stalled because of unexpected memory swapping or network errors.

Supervisor automatically reboots the server process if heartbeat breaks out.

se = ServerEngine.create(nil, MyWorker, {
  :daemonize => true,
  :pid_path => 'myserver.pid',
  :supervisor => true,  # enable supervisor process
})
se.run

You can restart a server process without waiting for completion of shutdown process (if supervisor and enable_detach parameters are enabled). This feature is useful to minimize downtime where workers take long time to complete tasks.

# 1. starts server
+------------+    +----------+    +-----------+
| Supervisor |----|  Server  |----| Worker(s) |
+------------+    +----------+    +-----------+

# 2. receives SIGINT and waits for shutdown of the server for server_detach_wait
+------------+    +----------+    +-----------+
| Supervisor |    |  Server  |----| Worker(s) |
+------------+    +----------+    +-----------+

# 3. starts new server if the server doesn't exit in server_detach_wait time
+------------+    +----------+    +-----------+
| Supervisor |\   |  Server  |----| Worker(s) |
+------------+ |  +----------+    +-----------+
               |
               |  +----------+    +-----------+
               \--|  Server  |----| Worker(s) |
                  +----------+    +-----------+

# 4. old server exits eventually
+------------+
| Supervisor |\
+------------+ |
               |
               |  +----------+    +-----------+
               \--|  Server  |----| Worker(s) |
                  +----------+    +-----------+

Note that network servers (which listen sockets) shouldn't use live restart because it causes "Address already in use" error. Instead, simply use worker_type=process configuration and send USR1 to restart only workers. USR1 signal doesn't restart server (by default. See also restart_server_process parameter). Restarting workers don't wait for completion of all running workers.

Robust servers should not restart only to update configuration parameters.

module MyWorker
  def initialize
    reload
  end

  def reload
    @message = config[:message] || "Awesome work!"
    @sleep = config[:sleep] || 1
  end

  def run
    until @stop
      logger.info @message
      sleep @sleep
    end
  end

  def stop
    @stop = true
  end
end

se = ServerEngine.create(nil, MyWorker) do
  YAML.load_file("config.yml").merge({
    :daemonize => true,
    :worker_type => 'process',
  })
end
se.run

Send USR2 signal to reload configuration file.

ServerEngine::BlockingFlag is recommended to stop workers because stop method is called by a different thread from the run thread.

module MyWorker
  def initialize
    @stop_flag = ServerEngine::BlockingFlag.new
  end

  def run
    until @stop_flag.wait_for_set(1.0)  # or @stop_flag.set?
      logger.info @message
    end
  end

  def stop
    @stop_flag.set!
  end
end

se = ServerEngine.create(nil, MyWorker) do
  YAML.load_file(config).merge({
    :daemonize => true,
    :worker_type => 'process'
  })
end
se.run

• interface
  • initialize is called in the parent process (or thread) in contrast to the other methods
  • before_fork is called before fork for each worker process (available only if worker_type is "process" or "thread")
  • run is the required method
  • stop is called when TERM signal is received
  • reload is called when USR2 signal is received
  • after_start is called after starting the worker process in the parent process (or thread) (available only if worker_type is "process" or "thread")
• api
  • server server instance
  • config configuration
  • logger logger
  • worker_id serial id of workers beginning from 0

• interface
  • initialize is called in the parent process in contrast to the other methods
  • before_run is called before starting workers
  • after_run is called before shutting down
  • after_start is called after starting the server process in the parent process (available only if supervisor parameter is true)
• hook points (call super in these methods)
  • reload_config
  • stop(stop_graceful)
  • restart(stop_graceful)
• api
  • config configuration
  • logger logger

ServerEngine supports 3 worker types:

• embedded: uses a thread to run worker module (default). This type doesn't support immediate shutdown or immediate restart.
• thread: uses threads to run worker modules. This type doesn't support immediate shutdown or immediate restart.
• process: uses processes to run worker modules. This type doesn't work on Win32 platform.

• TERM: graceful shutdown
• QUIT: immediate shutdown (available only when worker_type is "process")
• USR1: graceful restart
• HUP: immediate restart (available only when worker_type is "process")
• USR2: reload config file and reopen log file
• INT: detach process for live restarting (available only when supervisor and enable_detach parameters are true. otherwise graceful shutdown)
• CONT: dump stacktrace and memory information to /tmp/sigdump-.log file

Immediate shutdown and restart send SIGQUIT signal to worker processes which kills the processes. Graceful shutdown and restart call Worker#stop method and wait for completion of Worker#run method.

• Daemon
  • daemonize enables daemonize (default: false) (not dynamic reloadable)
  • pid_path sets the path to pid file (default: don't create pid file) (not dynamic reloadable)
  • supervisor enables supervisor if it's true (default: false) (not dynamic reloadable)
  • daemon_process_name changes process name ($0) of server or supervisor process (not dynamic reloadable)
  • chuser changes execution user (not dynamic reloadable)
  • chgroup changes execution group (not dynamic reloadable)
  • chumask changes umask (not dynamic reloadable)
  • daemonize_error_exit_code exit code when daemonize, changing user or changing group fails (default: 1) (not dynamic reloadable)
• Supervisor: available only when supervisor parameters is true
  • server_process_name changes process name ($0) of server process (not dynamic reloadable)
  • restart_server_process restarts server process when it receives USR1 or HUP signal. (default: false) (not dynamic reloadable)
  • enable_detach enables INT signal (default: true) (not dynamic reloadable)
  • exit_on_detach exits supervisor after detaching server process instead of restarting it (default: false) (not dynamic reloadable)
  • disable_reload disables USR2 signal (default: false) (not dynamic reloadable)
  • server_restart_wait sets wait time before restarting server after last restarting (default: 1.0)
  • server_detach_wait sets wait time before starting live restart (default: 10.0)
• Multithread server and multiprocess server: available only when worker_type is thread or process
  • workers sets number of workers (default: 1)
  • start_worker_delay sets wait time before starting a new worker (default: 0)
  • start_worker_delay_rand randomizes start_worker_delay at this ratio (default: 0.2)
• Multiprocess server: available only when worker_type is "process"
  • worker_process_name changes process name ($0) of workers
  • worker_heartbeat_interval sets interval of heartbeats in seconds (default: 1.0)
  • worker_heartbeat_timeout sets timeout of heartbeat in seconds (default: 180)
  • worker_graceful_kill_interval sets the first interval of TERM signals in seconds (default: 15)
  • worker_graceful_kill_interval_increment sets increment of TERM signal interval in seconds (default: 10)
  • worker_graceful_kill_timeout sets promotion timeout from TERM to QUIT signal in seconds. -1 means no timeout (default: 600)
  • worker_immediate_kill_interval sets the first interval of QUIT signals in seconds (default: 10)
  • worker_immediate_kill_interval_increment sets increment of QUIT signal interval in seconds (default: 10)
  • worker_immediate_kill_timeout sets promotion timeout from QUIT to KILL signal in seconds. -1 means no timeout (default: 600)
• Logger
  • log sets path to log file. Set "-" for STDOUT (default: STDERR)
  • log_level log level: debug, info, warn, error or fatal. (default: debug)
  • log_rotate_age generations to keep rotated log files (default: 5) (not dynamic reloadable)
  • log_rotate_size sets the size to rotate log files (default: 1048576) (not dynamic reloadable)
  • log_stdout hooks STDOUT to log file (default: true) (not dynamic reloadable)
  • log_stderr hooks STDERR to log file (default: true) (not dynamic reloadable)
  • logger_class class of the logger instance (default: ServerEngine::DaemonLogger)

Author:    Sadayuki Furuhashi
Copyright: Copyright (c) 2012-2013 Sadayuki Furuhashi
License:   Apache License, Version 2.0