kelvin@cluxnei:~/blog$ cat 2gb/02-the-process.md

2026-09-20 EN ~18 min read ler em português

I closed the terminal and the app died

Everyone hits this once. You ssh in, run npm start, open the IP in a browser, it works, you close the terminal, and the site goes down. The first instinct is to look for a way to keep the terminal open. That is the wrong path, and understanding why it is wrong is the switch that is missing for anyone who has only ever worked on a managed platform.

What actually killed your application

When you open an SSH session, the kernel creates a session with a controlling terminal. Everything you run there is born inside it. When the connection closes, the terminal is gone, and the kernel sends SIGHUP to the foreground process group. The default action for that signal is to terminate. Your npm start decided nothing. It received a signal saying the terminal that spawned it no longer exists.

The first instinct is to dodge the signal. nohup ignores SIGHUP, & backgrounds the job, tmux and screen keep a session alive on the other side. All three work, and none of them is the answer, because none of them addresses the questions that follow immediately:

• The machine rebooted overnight. Who starts the application again?
• The process died on an unhandled exception at four in the morning. Who notices?
• It leaked memory and took the database with it. Who was holding it back?
• You need the log from 2pm yesterday. Where is it?

Those four questions have a single answer, and it is already installed on your machine. systemd is PID 1: it started before everything else, it will keep running either way, and supervising processes is literally its job. I measured it in part 1: 6.4 MB of PSS. You are already paying for that supervisor whether you use it or not.

The minimum unit, line by line

One file at /etc/systemd/system/myapp.service. This is everything a Node application needs to become a real service:

[Unit] Description=my api After=network-online.target Wants=network-online.target [Service] Type=simple User=deploy Group=deploy WorkingDirectory=/srv/myapp/current EnvironmentFile=/srv/myapp/shared/.env ExecStart=/usr/bin/node /srv/myapp/current/server.js Restart=on-failure RestartSec=2 [Install] WantedBy=multi-user.target

systemctl daemon-reload systemctl enable --now myapp systemctl status myapp

enable is what makes it start at boot. --now starts it immediately as well. They are two separate things and it is worth knowing which is which, because forgetting the enable is the classic way to find out three months later that the machine does not come back on its own.

The four traps

ExecStart is not a shell. No pipes, no globs, no &&, and $VAR does not expand the way you expect. It is an absolute path and a list of arguments. If you genuinely need a shell, it is ExecStart=/bin/bash -lc '...', and at that point you have taken on a whole shell of overhead for a reason that is usually laziness.

The PATH is minimal. No ~/.nvm, none of what your .bashrc sets up. Use the binary's absolute path. With nvm the real path is something like /home/deploy/.nvm/versions/node/v22.22.0/bin/node, and finding that with which node in your interactive session does not help, because the service does not have your session.

User= is not optional. Without that line the service runs as root. A web application running as root turns any remote code execution into full control of the machine, and buys you nothing in return.

The .env is not a script. EnvironmentFile reads KEY=value per line, with no export, no magic quoting, no interpolating another variable. And it deserves chmod 600 with the right owner, because that is where the passwords live.

Coming back on its own, and the limit that stops the loop

Restart=on-failure starts the service again when the process exits non-zero or dies on a signal. There is also Restart=always, which restarts even after a clean exit, and that is what you want for a server that should never finish by itself.

I tested it with a service that prints its PID, waits two seconds and exits with an error, using RestartSec=1. Over eleven seconds:

subi, pid 2651864 subi, pid 2651915 subi, pid 2651938 NRestarts=3

A new PID each time round, and a counter exposed through systemctl show -p NRestarts, which is a good signal to monitor later.

Now the part that catches everyone the first time. If the application is genuinely broken it dies, starts, dies, starts. systemd has a brake for that: by default, five starts within ten seconds and it gives up, leaving the unit in failed state without trying again. The symptom is an application that stops coming back, with status saying start request repeated too quickly.

That is protection, not a bug. Tune it with StartLimitIntervalSec and StartLimitBurst, and clear the state with systemctl reset-failed myapp. A RestartSec of a few seconds is usually more useful than loosening the limit, because it gives the database time to come back before the next attempt.

Logs without a log file

Everything the process writes to stdout and stderr goes to journald. You create no file, configure no rotation, and run no risk of filling the disk with a 40 GB app.log, which is one of the most common ways to kill a small machine.

journalctl -u myapp -f # following journalctl -u myapp --since "1 hour ago" journalctl -u myapp -p err # errors and worse only journalctl -u myapp -n 200 --no-pager journalctl --disk-usage

The journal has a ceiling of its own, in /etc/systemd/journald.conf. On a small VPS, SystemMaxUse=200M is a sensible number, and it is one line that avoids a problem which only shows up months later.

One application choice that fits this: write one line per event, with no ASCII art and no colour. Colour in a terminal becomes an escape sequence in the journal, and then the grep you run at three in the morning finds nothing.

The memory ceiling, measured

This is the section that justifies all of part 2, and it is what makes a 2 GB VPS predictable instead of a gamble.

In part 1 we saw the global OOM killer: when RAM runs out, the kernel picks the largest process and kills it, which almost always means killing the database because something else leaked. The defence is giving every service a limit of its own, and cgroup v2 does that with two keys that behave very differently.

I set up the experiment with a program that allocates 200 MB and touches every page, so the memory is real rather than promised. I ran it three times, with systemd doing the measuring:

Read the middle row carefully, because it is the less known of the two. With MemoryHigh the program finished correctly. The kernel held resident memory at 64.2 MB and pushed 143.6 MB out to swap, charging twice the running time for it. MemoryHigh is a brake: cross the line and the process comes under reclaim pressure and gets slow, but stays alive.

MemoryMax is a wall. On touching it the kernel kills inside the cgroup, and the log says exactly that:

myapp.service: The kernel OOM killer killed some processes in this unit. myapp.service: Main process exited, code=killed, status=9/KILL myapp.service: Failed with result 'oom-kill'.

And in the kernel log, the word that changes everything:

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/system.slice/myapp.service Memory cgroup out of memory: Killed process 2650976 (python3) anon-rss:65296kB

CONSTRAINT_MEMCG means the out of memory condition was local to the cgroup, not to the machine. Compare it with the global message from part 1, which is just Out of memory: Killed process. The machine across the whole experiment: 4269 MB available before, 4257 MB after. It never found out.

That is the difference between a leak that takes down one service and a leak that takes down the whole site.

What numbers to use

The recipe that works is both keys together, with different jobs:

[Service] MemoryHigh=180M # brake: above this it starts to hurt MemoryMax=256M # wall: above this it dies on its own MemorySwapMax=256M

Put MemoryHigh a little above the measured normal usage, and MemoryMax comfortably above that. A legitimate traffic spike then gets slow and survives, and only a real leak reaches the wall. Measuring normal usage is systemctl show -p MemoryCurrent myapp or systemd-cgtop, remembering from part 1 that the number includes page cache.

This is not exotic configuration. polkit.service, installed by default on Ubuntu, already ships with MemoryMax=32M. The distribution uses the mechanism on its own services, while most applications people write run with no ceiling at all.

pm2, supervisor and friends

Anyone coming from Node usually reaches pm2 before reaching systemd. It works, the interface is pleasant, and there is a sum worth doing with real numbers.

I measured a real pm2 in production, counting only the supervision processes, with no application included:

On a 2 GB machine, 77 MB is nearly 4% of everything you have, spent on supervision. The sharpest detail in that table is the second row: a 40 MB Node process whose only job is rotating log files that journald would never have created.

Being fair to the tool, pm2 delivers things systemd does not give you for free: cluster mode with zero downtime reload, an ecosystem file describing several apps together, and pm2 monit. If you use those, pay the 77 MB knowing what you bought. If you use pm2 only so the process survives closing the terminal, you are paying 4% of the machine for something PID 1 already does.

The same reasoning covers supervisor, forever and the rest. The question is not which is better, it is what you are buying and what it costs in RAM on a machine where RAM is the wall.

Sidebar: PHP-FPM is a different model

If your application is PHP, almost none of the above applies the same way, because the supervisor comes built in. PHP-FPM has a master that forks children, and each child handles one request at a time. systemd supervises the master, and the master supervises the pool.

The consequence is that your concurrency limit is a number in a config file, and that number multiplies memory. With the 25 to 40 MB per child measured in part 1:

pm = dynamic pm.max_children = 8 # 8 x ~35 MB = ~280 MB at peak pm.start_servers = 2 pm.min_spare_servers = 1 pm.max_spare_servers = 3 pm.max_requests = 500 # recycle the child, cut off slow leaks

The factory default usually ships pm.max_children far too high for a small VPS. The sum is direct: multiply by the measured usage of one child and see whether it fits. If it does not, a traffic spike will not make the site slow, it will summon the OOM killer.

pm.max_requests deserves its own line. It kills and recreates the child after N requests, which turns a slow leak into a problem that never surfaces. It is a hack, and it is a hack that has worked for twenty years.

Where it listens

Your application has to accept connections somewhere, and there are two options.

A TCP port. The common way. The rule that matters: listen on 127.0.0.1:3000, never on 0.0.0.0:3000. With 0.0.0.0 the application is reachable directly on the public IP at port 3000, going around the reverse proxy, the TLS and any rule you put in front of it. The firewall from part 1 helps, and is not a reason to leave it open.

A unix socket. A file instead of a port, with filesystem permissions instead of firewall rules. It occupies no port, cannot be reached from outside by accident, and skips the whole TCP stack for traffic that never leaves the machine. It is what PHP-FPM uses by default on Ubuntu, at /run/php/php8.5-fpm.sock.

For an application behind a proxy on the same machine, a unix socket is the safer choice by construction. How nginx talks to each of the two is part 3.

The checklist

Your application is in order when all of these are true:

Seven command lines. If any of them fails, the problem is already written down on the server, waiting for the bad day.

What comes in part 3

The application runs, comes back on its own, has a ceiling and writes logs. And it is still unreachable from outside, because it listens on 127.0.0.1:3000 and nobody types port 3000 into a browser.

Part 3 is the front door: one IP, one port 443, and several applications behind it. Reverse proxy, the nginx matching order that is not the one you imagine, the headers that get lost on the way and make your application generate the wrong links, and where exactly TLS terminates.

kelvin@cluxnei:~/blog$