Last week, we moved one of our test servers to Nginx + Phusion Passenger Standalone 5 aka Raptor, and this is where systemd came in handy.
Updated on 2015-03-12: Provide “current” app directory as argument to fix restarts
2014 has seen systemd making its way into even more Linux distributions, including CentOS 7, which we’re using at trademate for running our Rails application servers.
While I’m seriously opposed to systemd’s “let’s do it all” architecture – I don’t want to be forced to use an init system which includes an HTTP server by default, and that’s just one of the many problematic design decisions of systemd (let’s skip the journal) – there’s also some good parts from an administrator’s point of view.
Systemd unit files support a thing called instantiation, which allows you to run multiple instances of the same service with different configurations. In our deployment, we want to run multiple Rails apps, each within its own Passenger Standalone instance. To get instantiation, the service unit file name must end with an “@” sign. The service can then be started multiple times by providing an extra parameter, the instance name, when enabling the service.
TL;DR
I know how to work unit files – just get me the code
Installing Passenger Standalone
Passenger can be installed in different ways. We’re using the classic gem method, but other methods should work as well.
# gem install passenger
This will provide you with the passenger-config commandline tool, which can be used to get the passenger installation root:
# passenger-config --root /opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2
Determining Rubygems environment
Use gem env to get the GEM_HOME environment variable, which may vary depending on your ruby installation:
# gem env RubyGems Environment: - RUBYGEMS VERSION: 2.2.2 - RUBY VERSION: 2.1.5 (2014-11-13 patchlevel 273) [x86_64-linux] - INSTALLATION DIRECTORY: /opt/rubies/2.1.5/lib/ruby/gems/2.1.0 [...]
In our case, GEM_HOME is /opt/rubies/2.1.5/lib/ruby/gems/2.1.0.
Deploying your Rails apps
All of your Rails apps should be deployed to some common location, each app occupying its own subfolder. For this guide, we’re going to use /apps to place our apps, so the application named demo would be deployed to /apps/demo, with its current release symlink at /apps/demo/current respectively.
Building the passenger systemd unit file
Using the following template, you can build your own passenger service unit file and place it in /etc/systemd/system/passenger@.service. Note the @ sign, which is important.
[Unit] Description=Passenger Standalone Application Server After=network.target [Service] Type=forking PrivateTmp=yes User=USERNAME Group=GROUPNAME WorkingDirectory=/apps/%i #RuntimeDirectory=passenger #RuntimeDirectoryMode=0755 PIDFile=/run/passenger/app.%i.pid Environment="PATH=/opt/rubies/2.1.5/bin:/usr/local/bin:/usr/bin" Environment="GEM_HOME=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0" Environment="GEM_PATH=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0" Environment="PASSENGER_INSTANCE_REGISTRY_DIR=/run/passenger" ExecStart=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2/bin/passenger start current --daemonize --instance-registry-dir /run/passenger --socket /run/passenger/app.%i.sock --pid-file /run/passenger/app.%i.pid --log-file /apps/%i/shared/log/passenger.log --environment production --max-pool-size=16 ExecReload=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2/bin/passenger-config restart-app /apps/%i ExecStop=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2/bin/passenger stop --pid-file /run/passenger/app.%i.pid [Install] WantedBy=multi-user.target
Depending on the Ruby deployment method you’re using, it may be necessary to change some of the paths and/or script names to use your version manager’s wrapper scripts.
- First, you should set the
UserandGroupoptions to the server user which you want to run the apps as. - The
WorkingDirectoryshould be set to the Rails app root. Systemd will replace%iwith the application name upon instantiation of the service. - The
Environmentoptions should be used to definePATH,GEM_HOMEandGEM_PATH. Insert the values from your local installation andgem env. - Replace the
passengerandpassenger-configbinaries’ path inExecStart,ExecReloadandExecStopwith the path provided bypassenger-config --root.
We’ll be using Nginx as SSL frontend and have it connect to the Passenger Standalone instances via UNIX sockets. This way, we can have a single UNIX socket per Rails app, which can be named like the Rails application, so we’re able to run multiple passenger instances without configuring a separate TCP listening port per instance.
Providing the service runtime directory
Passenger requires a filesystem location where it can place its PID files and the UNIX socket. By default, Passenger uses the Rails app’s tmp dir for its PIDs. This is problematic in conjunction with Capistrano (or any other deployment system which uses release symlinks), as after a deploy the tmp dir points to another location.
Therefore, we use the runtime directory functionality provided by systemd. Depending on your systemd version, two different approaches are required.
systemd >= 211
You’re lucky! Simply uncomment the RuntimeDirectory and RuntimeDirectoryMode options in your unit file. Systemd will automatically provide the runtime directory.
systemd < 211
CentOS 7 is currently using systemd 208, where the RuntimeDirectory options are not yet available. However, there’s systemd-tmpfiles which provides the same functionality. To enable it, create /etc/tmpfiles.d/passenger.conf with the following content:
#Type Path Mode UID GID Age Argument d /run/passenger 0755 username groupname
Of course, username and groupname must be replaced by the user Passenger is running as. After a restart, the /run/passenger dir will have been created. If you don’t want to reboot your machine, you can create the directory manually for now:
# mkdir /run/passenger # chown username:groupname /run/passenger # chmod 755 /run/passenger
The tmpfiles.d will ensure that it is recreated on reboot, as /run is a tmpfs filesystem.
Running multiple applications
This is where the magic happens. To start the demo application, run the following command:
# systemctl start passenger@demo
If everything went well, passenger should be up and running, which can be checked using
# systemctl status passenger@demo
If something goes wrong, check the journal:
# journalctl -xn
To stop the app:
# systemctl stop passenger@demo
To have the Rails application started on boot:
# systemctl enable passenger@demo
All of your other apps can be started and queried the same way, replacing the instance name demo with the app’s actual name. If you have a second app called foobar, you would enable it using
# systemctl enable passenger@foobar
Reloading
Passenger Standalone doesn’t support the classic touch restart.txt reloading method. To reload an application:
# systemctl reload passenger@demo
Alternatively, you can use the passenger-config tool for restarts while not root:
$ PASSENGER_INSTANCE_REGISTRY_DIR=/run/passenger passenger-config restart-app /apps/demo
Instance registry, passenger-config and passenger-status
Passenger uses the so-called instance registry directory to keep track of the application instances currently running. By default, /tmp is used. As the PrivateTmp option is enabled in the service file to improve security, Passenger management tools like passenger-status cannot access the instance registry any more. To avoid this, Passenger is configured to store its instance registry in the runtime directory provided by systemd.
Instance registry entries are named passenger.*. In order to prevent name clashes, per-app UNIX sockets and PID filenames are prefixed with app., so you shouldn’t run into problems when naming one of your apps passenger by coincidence.
To use passenger-status or passenger-config, you have to provide the PASSENGER_INSTANCE_REGISTRY_DIR env variable:
# PASSENGER_INSTANCE_REGISTRY_DIR=/run/passenger passenger-status
Version : 5.0.2
Date : 2015-01-23 20:38:19 +0100
Instance: HExxsQpf (Phusion_Passenger/5.0.2)
----------- General information -----------
Max pool size : 2
Processes : 1
Requests in top-level queue : 0
----------- Application groups -----------
/apps/demo/releases/20150121135417#default:
App root: /apps/demo/releases/20150121135417
Requests in queue: 0
* PID: 15101 Sessions: 0 Processed: 2 Uptime: 14m 5s
CPU: 0% Memory : 158M Last used: 14m 1s ago
Integrating with Nginx
Configuration is as usual, except for the backend server, which is accessed via UNIX socket instead of TCP:
location / {
proxy_pass http://unix:/run/passenger/app.demo.sock:;
}
The various applications can then be mounted on separate virtual servers.
Tuning
As you might have noticed, Passenger runtime options like MaxPoolSize are currently hard-coded in the unit file. While this is okay for our current requirements, you might want to customize some of the options on an per-application basis.
Adding a systemd environment file
Services can load environment variables from a file defined by the EnvironmentFile option.
[Service] EnvironmentFile=-/apps/%i/current/config/passenger.env
By prepending the path with a hyphen, you can tell systemd not to complain should the environment file not exist.
In the environment file, you can define well, … environment variables:
PASSENGER_OPTS="--max-pool-size 2 --min-instances 1"
These variables can then be supplied to the ExecStart et al. directives:
ExecStart=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2/bin/passenger start current --daemonize --instance-registry-dir /run/passenger --socket /run/passenger/app.%i.sock --pid-file /run/passenger/app.%i.pid --log-file /apps/%i/shared/log/passenger.log --environment production $PASSENGER_OPTS
This way, each app can have its individual configuration options, while all apps share one common unit file.
Running different Ruby versions… and tricking systemd
It’s even possible to customize the Ruby version on an per-app basis by moving PATH, GEM_HOME, GEM_PATH and PASSENGER_ROOT into the EnvironmentFile:
PATH="/opt/rubies/1.9.3-p194/bin:/usr/local/bin:/usr/bin" PASSENGER_ROOT="/opt/rubies/1.9.3-p194/lib/ruby/gems/1.9.1/gems/passenger-4.0.24" GEM_HOME="/opt/rubies/1.9.3-p194/lib/ruby/gems/1.9.1" GEM_PATH="/opt/rubies/1.9.3-p194/lib/ruby/gems/1.9.1"
Variables defined by EnvironmentFile will override variables defined by Environment.
However, systemd doesn’t allow environment variable interpolation in the actual ExecStart command, which is required to be an absolute path. Environment variables may only be used in the arguments following the command. By wrapping the commandline into a bash exec call, we get env variable interpolation back and can set the executable paths dynamically through the EnvironmentFile.
[Service]
Environment="PATH=/opt/rubies/2.1.5/bin:/usr/local/bin:/usr/bin"
Environment="GEM_HOME=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0"
Environment="GEM_PATH=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0"
Environment="PASSENGER_ROOT=/opt/rubies/2.1.5/lib/ruby/gems/2.1.0/gems/passenger-5.0.2"
Environment="PASSENGER_INSTANCE_REGISTRY_DIR=/run/passenger"
EnvironmentFile=-/apps/%i/current/config/passenger.env
ExecStart=/usr/bin/bash -c 'exec ${PASSENGER_ROOT}/bin/passenger start current --daemonize --instance-registry-dir /run/passenger --socket /run/passenger/app.%i.sock --pid-file /run/passenger/app.%i.pid --log-file /apps/%i/shared/log/passenger.log --environment production $PASSENGER_OPTS'
ExecReload=/usr/bin/bash -c 'exec ${PASSENGER_ROOT}/bin/passenger-config restart-app /apps/%i'
ExecStop=/usr/bin/bash -c 'exec ${PASSENGER_ROOT}/bin/passenger stop --pid-file /run/passenger/app.%i.pid'
Code
A unit file template which integrates all of the above features can be downloaded at https://github.com/mtgrosser/passenger-systemd.