Hosting a Phoenix Web App on OpenBSD

I have some Phoenix web apps and wanted to host them on OpenBSD. It's just a few small apps on a single server, not anything complex, so I was hoping to figure out a simple approach (no need for anything enterprise-grade or web-scale). I think I was able to come up with something good, which I'll share below.

Packet filter

I planned on running my web apps in distributed mode (with the --sname option to elixir). In distributed mode, the Erlang VM will listen on a non-privileged port on all addresses and will register with the Erlang port mapping daemon which also (by default) listens on a non-privileged port on all addresses. I found that the port mapper can be configured to listen only on the loopback interface, but I didn't find a way to have the Erlang VM do the same.

In any event, I didn't plan on having regular user processes listen on public interfaces so I added a rule to pf.conf(5) to block incoming packets to non-privileged ports being listened on by regular users.

block return in on ! lo0 proto { tcp, udp } to port 1025:65535 user >= 1000

After adding the rule, I reloaded the packet filter configuration (as root).

pfctl -f /etc/pf.conf

Installing Elixir and Erlang

Phoenix is written in Elixir, which in turn requires Erlang. I checked the Elixir install guide and found that there's an OpenBSD package for Elixir.

pkg_add elixir

Following the suggestion given by the installer, I created /etc/man.conf and added Erlang's manual page directory in case I should want to examine those manual pages. The manual page for man.conf provided the defaults.

manpath /usr/share/man
manpath /usr/X11R6/man
manpath /usr/local/man
manpath /usr/local/lib/erlang21/man

I added a line to my ~/.profile to set LC_ALL. Setting LC_ALL is important on OpenBSD because although the base system mostly ignores the locale (other than having limited support for UTF-8), it is very important to Elixir that the Erlang VM run with UTF-8 character encoding (Elixir uses UTF-8 natively, but depends on support from the VM to do so). The locale(1) manual page has more details about OpenBSD locale support.

export LC_ALL="en_US.UTF-8"

I also pre-emptively installed Hex (the package manager for Elixir and Erlang) and rebar (an older build tool for Erlang, used by a Phoenix dependency) since all of my web apps use them.

mix local.hex --force
mix local.rebar --force

Installing postgres

As is common, my Phoenix web apps use Ecto and PostgreSQL for persistence. I installed the PostgreSQL client and server packages.
pkg_add postgresql-client postgresql-server 

After installing the packages, I initialized a database cluster as the _postgresql user. I used the flags recommended in /usr/local/share/doc/pkg-readmes/postgresql-server. They set the PostgreSQL super-user user name to “postgres”, enable scram-sha-256 authentication (current best-practice — the likely alternative being md5, which is deprecated), set the encoding to UTF-8, and causes the initdb program to prompt for a new super-user password.

su - _postgresql
mkdir /var/postgresql/data
initdb -D /var/postgresql/data -U postgres -A scram-sha-256 -E UTF8 -W
exit

There was some performance tuning guidance in the read-me file too. I glanced through it. It may come in handy if I need to scale my apps up in the future.

I enabled the postgresql daemon and started it.

rcctl enable postgresql
rcctl start postgresql

I created a new PostgreSQL user and a new database for one of my web apps. I'll need to do this for each web app. The user name, password, and database name become part of DATABASE_URL environment variable I'll specify when running the web app. The new user is set up as the owner of the new database.

createuser -P -U postgres hitcounter
createdb -O hitcounter -U postgres hitcounter

Backups with Tarsnap

I already had an account with Tarsnap from other projects and I thought it would be nice to configure this VM to use it for automatic backups. I chose to build it from source, following the compilation instructions.

I'm only interested in backing up my PostgreSQL databases since that's where all of my web app data lives. I already have the application source code and the system configuration is reasonably well documented in these articles. Because of this simple use-case, I won't need to run tarsnap with root permissions. So, I installed it in my user's home directory rather than in /usr/local.

tar -xzf tarsnap-autoconf-1.0.39.tgz
cd tarsnap-autoconf-1.0.39
./configure --prefix=/home/user/tarsnap
make all
make install

Continuing with the getting started instructions, I registered my machine and made a key file. It's important to keep a copy of the key someplace safe. I use macOS locally, so I put it in my login keychain as a secure note using Keychain Access.

tarsnap/bin/tarsnap-keygen --keyfile tarsnap.key --user xxxxxx --machine xxxxxx

I put together a script to dump the PostgreSQL database for my web app into a staging directory, then back up the staging directory with Tarsnap. After that, it lists the backups in Tarsnap and deletes all but the 90 most recent. I plan to run this script daily, so that should give me 90 days of backups. The script includes a user name and password for the web app database, so it's important that the script file be readable only by my user (mode 0700). If I add more web apps, I'll need to add lines to this script for each of their databases.

#!/bin/ksh
PGPASSWORD=xxxxxxxx pg_dump -f backup-staging/hitcounter.sql \
	-U hitcounter hitcounter
tarsnap/bin/tarsnap -c --keyfile tarsnap.key --cachedir tarsnap/cache \
	-f $(date +%s) backup-staging
tarsnap/bin/tarsnap --list-archives --keyfile tarsnap.key | sort -nr | \
                sed '1,90d' | while read name ; do
        tarsnap/bin/tarsnap -d --keyfile tarsnap.key --cachedir tarsnap/cache \
                -f $name
done

I wanted to run the script every day, so I added it to my user's crontab(5).

@daily ~/backup.sh

Prepare the web app

At least at the time I was setting this up (and perhaps still), the default endpoint settings in the generated config/prod.secret.exs would cause the web app to listen on all IPv6 addresses. I preferred that it listen only on the local interface since I didn't want it accessible to the outside world except through relayd and that is where relayd would look for it. I updated the relevant portion of config/prod.secret.exs in my app.

config :hitcounter, HitcounterWeb.Endpoint,
  http: [:inet, ip: {127, 0, 0, 1}, 
    port: String.to_integer(System.get_env("PORT") || "4000")],
  secret_key_base: secret_key_base

It's often necessary for an app to generate a full URL that points to the app itself, so it needs to know the base URL to use. I updated the Endpoint configuration my app's config/prod.exs with the correct information. In particular, I added the correct host name, scheme, and port. Because the app is running behind relayd, these are all different than what Phoenix knows about.

config :hitcounter, HitcounterWeb.Endpoint,
  url: [host: "hitcounter.parksdigital.com", scheme: "https", port: 443],
  cache_static_manifest: "priv/static/cache_manifest.json"

Running the web app

I looked at the process for building and deploying Phoenix web apps using releases and it seemed pretty great. But, I think — at least at this time — that I don't need most of what it provides. I decided to just rsync the app to the server and run it with the environment set to &ldquot;prod&rdquot;.

Compilation (particularly of dependencies) takes a little while the first time, but Elixir is smart about not re-compiling things that haven't changed so it's not too bad after that.

I wanted to exclude some files from the rsync, which was not (yet?) supported by openrsync, so I installed rsync.

pkg_add rsync

From my local development machine I rsync'd the app up to the OpenBSD server, excluding anything that was in .gitigore (build artifacts, dependencies, and such) as well as the .git directory.

rsync -r --delete --exclude-from .gitignore --exclude .git -l . vultr:hitcounter

I wanted the web app to keep running after I logged out, so I put together a script to make a tmux session and start the app in one of its windows. The script sets several environment variables for the app: production mode, database connection details, secret key base, and port to listen on. Then it gets dependencies, compresses and digests static assets, creates and initializes the database (we've already created the database and the PostgreSQL user we made doesn't have permission to create it anyway, so this will produce an error but it also initializes the table Ecto uses for tracking migrations if it doesn't exist, which is important), performs any necessary database migrations, and then starts the app in distributed mode (so an interactive Elixir session can be attached to it).

#!/bin/ksh
app_name=hitcounter
db_pass=xxxxxxxxxxx
secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
cd ~/$app_name
tmux new-session -d -n shell -s $app_name
tmux set-option -g -t $app_name remain-on-exit on
tmux new-window -d -n server -t $app_name:1 /bin/ksh -l -c "\
        export MIX_ENV=prod; \
        export DATABASE_URL=ecto://$app_name:$db_pass@localhost/$app_name; \
        export SECRET_KEY_BASE=$secret; \
        export PORT=4200; \
        mix deps.get; \
        mix phx.digest; \
        mix ecto.create; \
        mix ecto.migrate; \
        elixir --sname $app_name -S mix phx.server"
tmux new-window -d -n iex -t $app_name:2 /bin/ksh -l -c "\
        iex --sname $app_name-iex --remsh $app_name"
tmux new-window -d -n psql -t $app_name:3 \
        "PGPASSWORD=$db_pass psql $app_name $app_name"

I made sure to chmod 0700 this script too since it contains secrets.

After starting the app, the script goes on to add a few other useful windows in the tmux session: one with an interactive Elixir session attached to the running app and one with psql logged into the app's database as the app. There's also a window with a shell in the app's directory from when the session was created.

If I need to, I can log in to the server and attach tmux to this session and have everything I need to diagnose or debug the app at my fingertips.

tmux attach -t hitcounter

If I want to re-start the application (perhaps after rsync'ing over a new version), I can do so while attached to the tmux session and looking at the app server window by hitting control-B followed by colon to get a command prompt, then entering respawn-window -k. This works in the interactive Elixir window too, in case it becomes disconnected. The re-spawn can be initiated from the command line as well.

tmux respawn-window -t hitcounter:1 -k

To stop the entire app, I can kill the tmux session.

tmux kill-session -t hitcounter

I wanted my apps to start up with the system, too, so I added a line to my user's crontab to run the script at startup.

@reboot ~/hitcounter.sh

Deployment script

Deployment wasn't too bad with this setup, but I thought I could automate it with a script inside my application's project directory.

#!/bin/bash
app_name=hitcounter
rsync -r --delete --exclude-from .gitignore --exclude .git -l \
	. vultr:$app_name
ssh vultr "tmux respawn-window -t $app_name:1 -k"

Now, after making changes to the app locally, I can call this script to deploy it and have the changes live in a few seconds.

Wrapping up

I hope that this article has been helpful to you. If this is the kind of thing you're into, you may enjoy my other work. If you have any questions or comments, please feel free to drop me an e-mail.

Aaron D. Parks
Parks Digital LLC
4784 Pine Hill Drive, Potterville, Michigan
support@parksdigital.com

hit counter