kelvin@cluxnei:~/blog$ cat 2gb/01-the-machine.md
How much fits in 2 GB
You know the difference between Fargate and Cloud Run. You know when a managed queue is the right call and when it is overkill. You can estimate an egress bill. And you have never opened a terminal on a machine that is yours, with no console, no dashboard, and nobody guaranteeing it stays up. This series takes the cheapest VPS you can rent and puts real things on it. The first step is not installing anything. It is learning to see the machine.
In this text
A VPS is not a service
Here is what you are renting: a number of vCPUs, an amount of RAM, a disk, and a monthly transfer allowance. No health check. No autoscaling. Nobody restarting anything. If the process dies at three in the morning, it stays dead until you look.
In exchange you get the other side of that coin, which is exactly what disappears when everything is managed: there is no cold start, no per invocation billing, no surprise egress bill, and the whole machine is yours all the time, including while it sits idle.
Of the four resources, one dominates the rest. CPU you wait for. Disk you buy. Transfer you optimise. RAM is a wall. When it runs out the kernel does not warn you and does not degrade gracefully: it picks a process and kills it. That is why this entire first part is about memory.
The first ten minutes
You have an IP and a root password. Before installing anything there is a short sequence that separates a machine that is yours from a machine that will belong to somebody else in two weeks. It really is short:
# a user that is not root adduser deploy usermod -aG sudo deploy # your key, and only your key ssh-copy-id deploy@YOUR_IP # in /etc/ssh/sshd_config.d/99-hardening.conf PermitRootLogin no PasswordAuthentication no KbdInteractiveAuthentication no systemctl restart ssh # firewall: deny everything, allow what you use ufw default deny incoming ufw allow OpenSSH ufw allow 80,443/tcp ufw enable # security patches on their own apt install unattended-upgrades dpkg-reconfigure --priority=low unattended-upgrades
One note about the order, because the trap is a classic: test the key in a
second SSH session before turning passwords off. Keep the current terminal
open. If the key does not work and you have already restarted sshd
with PasswordAuthentication no, your only way back in is the
provider's rescue console, which is always worse than it sounds.
This is a minimum, not security. The part that holds up over time, meaning log rotation, kernel updates and what actually gets attacked on an exposed machine, is part 5.
You are reading free -h wrong
This is the moment where almost everyone coming from a managed platform concludes that 2 GB is useless. You bring the server up, run the command, and see this:
total used free shared buff/cache available Mem: 1971 612 158 14 1201 1203
One hundred and fifty-eight megabytes free. The natural reading is panic, and the natural reading is wrong.
Linux uses every spare byte of RAM as disk cache, deliberately. Every file
read stays in memory in case it is read again. Free RAM is wasted RAM.
That 1201 under
buff/cache is files, and files can be dropped the instant somebody
needs the space, losing nothing, because the authoritative copy is on disk.
The column that answers the question you are actually asking is called
available: how much memory a new process could ask for right now,
already counting what the kernel would hand back if it had to. That is 1203 MB,
not 158.
The rule: read available, never
free. If available is comfortable the machine is fine,
whatever free says.
Anonymous memory versus page cache
It is worth separating the two kinds, because the distinction explains almost everything that follows:
- Anonymous memory is what the process asked for and filled: heap,
stack, whatever came out of
malloc. It has no copy on disk. If the kernel wants that space back, its only option is swap. - Page cache is memory mirroring a file. It has a disk copy by definition. The kernel throws it away whenever it likes, for free.
When you look at "how much is this service using", you are almost always
looking at the sum of the two. And the gap between them is brutal. These are four
real services, measured through the cgroup's memory.stat:
| service | anonymous (real) | page cache | "used" |
|---|---|---|---|
| fail2ban | 15.5 MB | 293.1 MB | 309.8 MB |
| systemd-journald | 0.6 MB | 66.1 MB | 68.9 MB |
| php-fpm | 33.9 MB | 41.4 MB | 77.3 MB |
| cron | 0.0 MB | 19.3 MB | 20.7 MB |
fail2ban shows up on any dashboard eating 310 MB. It uses 15.5. The other 293 are log files it read, which the kernel decided to keep in memory because there was room to spare. The day there is not, they evaporate and nobody notices.
cron is the extreme case: zero memory of its own, 19 MB of
cached files. It "uses" 20 MB and uses nothing.
To see this for any service on your machine:
cat /sys/fs/cgroup/system.slice/name.service/memory.stat # the first two lines are anon and file
And top lies too
With the cache sorted out there is still a counting error left, and this one
catches experienced people. top, htop and
ps report RSS, every page of physical memory the process has
mapped. The problem: when five processes share the same libc, the same OpenSSL
and the same binary, RSS counts that single copy five times.
A real nginx, one master and four workers:
PID CMD PSS RSS 1656900 nginx: master process 1.7 10.5 1656903 nginx: worker process 3.7 12.7 1656904 nginx: worker process 3.6 12.6 1656905 nginx: worker process 3.5 12.5 1656907 nginx: worker process 3.6 12.6 sum 16.1 61.0
Adding up RSS gives 61 MB. Real usage is 16.1 MB. Those 45 MB of difference are the same memory counted again for every process.
The honest metric is PSS, proportional set size: every shared page is divided by the number of processes mapping it. If four workers share 8 MB of libraries, each one carries 2 MB on the books. The sum of PSS across every process on the machine matches the memory actually occupied, which the sum of RSS never does.
awk '/^Pss:/{s+=$2} END{printf "%.1f MB\n", s/1024}' /proc/PID/smaps_rollup
This matters more than it looks the moment you size anything that works by
forking: php-fpm with twenty children, gunicorn with eight workers, Puma
in cluster mode. Taking one child's RSS from htop and multiplying by
the child count overestimates the bill by a wide margin, and makes you buy a
machine you did not need.
Swap is not a defeat
There is a superstition that enabling swap admits the machine is too small. The opposite is true: on a 2 GB VPS, swap is what turns an OOM kill at three in the morning into two hundred milliseconds of slowness.
Swap is borrowed disk for anonymous pages, which are precisely the ones the kernel cannot discard for free. And a good share of a server's anonymous memory is cold: startup code from a daemon that ran once at boot, the admin tool you used last week, tables a service built and never touched again. Pushing that to disk is a straight win, because the freed RAM becomes cache for things actually in use.
Two misunderstandings worth clearing up:
vm.swappiness is not "percentage of RAM used before swapping
starts". It is the weight the kernel gives to evicting an anonymous page versus
evicting page cache when it needs memory back. The default is 60. On an SSD that
is perfectly reasonable, and touching that number should be one of the last
things you do.
Swap existing and being occupied is not a problem. What matters is the
flow, not the stock. Watch it in vmstat 1, columns
si and so, swap in and out per second:
vmstat 1 # si/so sitting at 0 with swap occupied → healthy, cold pages # si/so hammering non stop → you are genuinely out of RAM
On a 2 GB machine a 2 GB swapfile is a sensible default, and it does not need to be a partition:
fallocate -l 2G /swapfile chmod 600 /swapfile mkswap /swapfile swapon /swapfile echo '/swapfile none swap sw 0 0' >> /etc/fstab
Load average is not a percentage
The three numbers in uptime are averages over one, five and
fifteen minutes. They do not run from 0 to 100 and they have no direct relation
to CPU usage.
A load of 1.00 on a 1 vCPU machine means exactly busy. A load of 1.00 on a 4
vCPU machine means a quarter busy. The number only means something divided by
nproc.
On Linux there is one extra detail: the count includes processes in
uninterruptible sleep, which is typically waiting on disk. A sick disk raises
load while the CPU does nothing at all. If load is high and top
shows idle CPU, your bottleneck is I/O, not processing.
The three values together are a derivative. One minute far above fifteen minutes means something started recently and is getting worse. The reverse means the peak has passed.
The OOM killer kills the wrong thing
When a process asks for anonymous memory, there is nothing left to reclaim and swap is gone, the kernel goes out of memory. It cannot return an error to everyone, so it picks a process and kills it.
The pick is by oom_score, roughly proportional to the memory the
process occupies. Which means: it kills the biggest one.
And the biggest one is almost never the script that leaked. It is your database, or your main application.
This is the symptom that shows up on forums as "my app just died on its own and there is nothing in the log". There is nothing in the application log because the application did not decide to die. The one who knows what happened is the kernel:
journalctl -k | grep -i "killed process" # Out of memory: Killed process 4127 (postgres) total-vm:...
The fix is not buying RAM. The fix is making it impossible for one process to take the whole machine down, by putting a ceiling on every service, and that is exactly what part 2 does with cgroups. The goal is that the leak dies and nothing dies with it.
The budget: what each piece costs
Now that the numbers mean something, we can build the sum this series is named after. Everything below is real memory, measured with the process idle and under no load:
| piece | real memory |
|---|---|
| Ubuntu 26.04 LTS, 16 stock services | 24.8 MB |
| nginx, 1 master + 4 workers | 16.1 MB |
| PostgreSQL 17, idle | 19.6 MB |
| Redis 7, idle | 20.2 MB |
| PHP-FPM, master | 14.1 MB |
| PHP-FPM, per pool child | 25 to 40 MB |
| Node.js, application process | 48 to 96 MB |
Four readings of that table, in order of surprise:
Ubuntu costs 25 MB. Not 500. All sixteen stock services put together, including SSH, systemd, networking, DNS, clock and automatic updates, fit in twenty-five megabytes of their own memory. Its reputation for being heavy comes from disk and from the size of the installed package set. At rest, RAM has nothing to do with it.
nginx costs 16 MB. And inside those 16 MB it handles thousands of simultaneous connections, because it does not use a thread per connection. That model is the subject of part 3, and it is half the explanation for why the 2 GB sum works out.
Idle Postgres costs 20 MB, the number almost nobody believes. But
Postgres has a second number, and that is the one that kills small machines. Each
connection is a separate process. Measured: the server went from 19.6 MB to 94.3
MB on opening thirty idle connections, which is roughly 2.5 MB per
connection doing nothing at all. The factory default is
max_connections = 100. That is 250 MB of parked processes before a
single query, and before the work_mem every sort reserves on top. It
is the number one cause of death by Postgres on a small VPS, and the tuning is in
part 4.
Node is the expensive one. A typical application process measured between 48 and 96 MB, and a heavier one came in at 624 MB by itself. A single Node process can cost as much as forty nginx workers. If you are coming from serverless, this is the number that changes your planning: the cost is the runtime, not the framework.
Closing the sum
With those numbers, a full stack in 2 GB looks like this:
2048 MB total on the machine - 150 system and kernel structures (reserve) - 16 nginx - 20 idle PostgreSQL - 150 60 database connections, worst case - 20 Redis - 300 3 Node applications at ~100 MB ──────── 656 MB left for cache and for peaks
Reverse proxy, database, in-memory cache and three applications. On the cheapest VPS there is, with a third of the memory still free.
The 150 MB reserve for the kernel is the only figure in this sum that is not a direct measurement. Kernel structures (slab, page tables, network buffers) scale with what the machine does, so measuring them on one box and applying them to another would mean nothing. I treated it as a reserve, and a generous one.
None of this is a machine serving ten thousand requests per second. It is a machine doing what a small product does. And that is the whole point of the series: the distance between "I have nothing" and "I need a cluster" is much longer than the service catalogue lets you imagine.
The six commands
To close, the minimum toolbox. With these six you can answer "what is this machine doing right now", which is the question the managed platform had been answering for you:
| command | the question it answers |
|---|---|
free -h | do I have memory? read available |
vmstat 1 | am I really under pressure? columns si and so |
systemd-cgtop | who is spending it, per service |
df -h and du -xh -d1 / | the disk, which is the other wall |
journalctl -u X -f | what is this service saying |
ss -tulpn | who is listening, on which port |
systemd-cgtop is the most underrated one on the list. It reports
aggregate usage per service rather than per process, which on its own solves the
problem of adding up php-fpm children or nginx workers by hand.
What comes in part 2
You can read the machine now. It still runs nothing of yours, and the first obstacle is more humbling than it sounds: you start the application, close the terminal, and it dies with it.
Part 2 is about the switch that is missing for anyone coming from a platform: your application is not a deploy, it is a process. A process that has to survive a logout, come back on its own after a crash, start at boot, write logs somewhere findable, and above all have a memory ceiling, so that the day it leaks is the day only it dies.
kelvin@cluxnei:~/blog$