kelvin@cluxnei:~/blog$ cat 2gb/05-keeping-it-up.md

2026-09-20 EN ~22 min read ler em português

The part that only hurts later

Everything is up and you can publish. The four previous parts were about getting things standing. This one is about them staying up while nobody is watching, and it is what separates a project that lasts a year from one that dies on a Sunday at three in the morning for a reason that looked too small to deserve attention.

Backups you have restored

I will start with the one sentence in this part worth more than all the others: a backup you have not restored is not a backup, it is hope.

Everyone sets up a pg_dump in a cron job and marks the task done. The file gets written, grows every day, and nobody has ever tried to open it. The ways that can be broken the whole time without anyone noticing are many and all silent: the dump came out empty because the password changed, came out truncated because the disk filled halfway, came from the wrong database, or sits there perfectly and the restore command fails because the Postgres version on the new server is different.

The test is to go all the way through, and it takes fifteen minutes:

# 1. restore into a throwaway database, on your machine createdb restore_test pg_restore -d restore_test backup-2026-09-19.dump # 2. check there is data, not just a schema psql restore_test -c "select count(*) from users;" # 3. point the application at it and open a screen DATABASE_URL=postgres://localhost/restore_test npm start # 4. only now does the backup exist dropdb restore_test

Three rules about where it lives:

• Off the machine. A backup on the same disk protects against DROP TABLE and against nothing else. The disk dying, the provider disappearing and the account being suspended all take both at once.
• Outside your provider account if you can. An account suspended by mistake is more common than a disk failing.
• With a date in the name and a decided retention. Seven daily, four weekly. With no retention, the backup becomes the cause of the full disk in the next section.

Three things need backing up, and the second and third are the forgotten ones: the database, part 4's shared/ with the uploads and the .env, and the system configuration you wrote by hand, meaning the nginx files, the systemd units and the crontab. That last group fits in a private repository and saves a whole day when you have to rebuild the machine from scratch.

A full disk does not announce itself properly

Part 1 said RAM is the wall. Disk is the other one, and it is worse to diagnose, because there is no disk full killer that kills a process and leaves a line in the kernel log. What there is instead is everything failing differently at the same time.

I mounted a 56 MB filesystem and put a process writing fixed size records into it, the way a service would, until the space ran out:

the writer died on record 1394638: [Errno 28] No space left on device # the last two lines of the file: 01394637 complete application log record 01_ <- partial write

Look at the last line. It is not missing, it is half written. The process asked to write a whole record, the kernel wrote what fitted and returned an error. Anything reading that file later finds a corrupt record, and that is precisely how a full disk contaminates data rather than simply stopping a service.

And the behaviour that follows is what confuses people most. With the filesystem reading 100% and zero free blocks, I tried three writes of different sizes:

df: 55M used, 0 free (100%) inodes: 13 used, 16,371 free (1%) echo test > new.txt -> worked, 6 bytes written touch empty.txt -> worked, 0 bytes dd of 1 MB -> failed: No space left on device

A full disk does not fail uniformly, it fails by size. The small write gets through, the large one does not. That is why the symptoms never add up: your service goes on writing log lines, the config file you edited saves fine, and the database checkpoint fails. You conclude the problem is the database.

ext4 reserves 5% of blocks for root. That is why, with df reading 100%, you can still SSH in and fix things: your root session has a reserve that the services do not. That reserve is the only reason a full disk is recoverable remotely, and it is there by default without anyone asking.

Log rotation, and the journal nobody capped

The number one cause of a full disk on a small machine is logs. And the most honest example was right in front of me: I went to measure journald on an Ubuntu server in real use and found this:

journalctl --disk-usage 432.7M grep SystemMaxUse /etc/systemd/journald.conf (nothing. The default is 10% of /var)

Four hundred and thirty two megabytes of logs, because nobody set a ceiling. On a machine with a big disk that goes unnoticed forever. On a 2 GB VPS with a 25 GB disk, the 10% default authorises the journal to reach 2.5 GB before it starts worrying.

One line fixes it, and it is the same one from part 2:

# /etc/systemd/journald.conf SystemMaxUse=200M SystemMaxFileSize=20M systemctl restart systemd-journald journalctl --vacuum-size=200M # clears what is already there

For whatever does not go through the journal there is logrotate. The classic candidate is the nginx access_log, which on a small VPS grows faster than anything else:

# /etc/logrotate.d/myapp /srv/myapp/shared/logs/*.log { daily rotate 7 compress delaycompress notifempty missingok copytruncate }

copytruncate deserves an explanation, because it is the difference between working and not working. logrotate normally renames the file and waits for the service to reopen the new one. An application that does not know how to do that keeps writing into the renamed file, which goes on occupying disk and disappears from your tail. With copytruncate it copies and empties the original, the open descriptor stays valid, and the service never notices.

And the cheapest tuning of all, which part 3 already used: access_log off on the static file location. That is most of a site's log lines, and none of them tell you anything.

The run that starts before the previous one finished

You schedule a sync every five minutes. One day it takes six. From then on you have two copies running together, then three, each one consuming part 1's memory on a machine with no room for two.

I measured with a task taking five seconds, fired every two:

flock -n tries to take a lock file and, failing that, gives up silently. Six of the ten firings simply did not run, which is exactly the desired behaviour for a task that must not overlap:

*/5 * * * * /usr/bin/flock -n /tmp/sync.lock /srv/myapp/bin/sync.sh

There is a better alternative, and it is already installed. A systemd timer does not overlap by construction, because a unit cannot be active twice:

# myapp-sync.timer [Timer] OnCalendar=*:0/5 Persistent=true RandomizedDelaySec=30

Three advantages over cron, and all of them matter on a small box. Persistent=true runs the missed job once the machine comes back. RandomizedDelaySec spreads the firing out, so five tasks scheduled for 0 3 * * * do not wake in the same second and build a memory spike at three in the morning. And the service a timer fires is an ordinary unit, so it inherits part 2's MemoryMax and journal logging for free.

I published and still see the old thing

That sentence has four suspects, and they have an order. From closest to the user to closest to your code:

1. browser cache whatever you sent in Cache-Control 2. CDN or outer proxy may ignore what you sent 3. nginx cache proxy_cache, open_file_cache 4. application cache route, view, query, opcode

Item 2 is the treacherous one, because it contradicts the intuition that the origin is in charge. I built the demonstration with a backend answering the current time with Cache-Control: no-store, no-cache, must-revalidate, behind two identical caches, changing only whether they respect that header:

# cache respecting the origin version generated at 21:22:35 X-Cache: MISS version generated at 21:22:36 X-Cache: MISS version generated at 21:22:38 X-Cache: MISS # same backend, cache configured to ignore version generated at 21:22:39 X-Cache: HIT version generated at 21:22:39 X-Cache: HIT version generated at 21:22:39 X-Cache: HIT

The clock froze. The backend kept generating the right time and shouting no-store on every response, and the cache in front decided it was none of its business. One configuration line, proxy_ignore_headers Cache-Control, and the origin lost authority over its own content.

It is the same mechanism as a CDN configured to override the origin's TTL, and it is why I published and still see the old thing is so hard to debug: you look at your code, you look at your headers, both are right, and the wrong answer comes from a layer you had forgotten existed.

The order for debugging, always inside out, because checking from outside gives you an answer that went through every layer:

curl -s localhost:3000/route # 4. is the application right? curl -s -H 'Host: mysite.dev' localhost/route # 3. and nginx? curl -s https://mysite.dev/route # 2. and the outer layer? # 1. the browser: private window, or Ctrl+Shift+R

The first line where the content goes wrong is the guilty layer. And the prevention, which part 3 already mentioned, is putting a hash in the file names: a new name is never cached anywhere, and the problem stops existing rather than being solved.

Monitoring that fits

The temptation here is to assemble the full stack. I measured what it costs idle, collecting almost nothing:

That is 22% of a 2 GB machine to observe an application which, by part 1's numbers, occupies around 100 MB. Grafana alone costs more than three Node processes. On a box that size, this stack does not observe the problem, it is the problem.

What genuinely matters at this scale is five things, and four of them are free:

• An external uptime check. The only one that has to live off the machine, because monitoring that runs on the server cannot tell you when the server is down. A free service, a ping every minute, an email alert.
• Disk. An alert at 80% gives you days of warning. It is the highest return item on the whole list.
• Available memory, part 1's available column, not free.
• Service state. systemctl is-failed on each unit that matters.
• Certificate expiry. Part 3's automatic renewal fails silently, and you find out ninety days later.

That fits in one script and one cron entry, and occupies zero bytes of memory at rest:

#!/usr/bin/env bash alert(){ echo "$1" | mail -s "[server] $1" you@example.com; } used=$(df --output=pcent / | tail -1 | tr -dc 0-9) [ "$used" -ge 80 ] && alert "disk at ${used}%" free=$(free -m | awk 'NR==2{print $7}') [ "$free" -lt 200 ] && alert "available memory: ${free} MB" for u in myapp nginx postgresql; do systemctl is-failed --quiet "$u" && alert "$u is down" done days=$(( ($(date -d "$(openssl s_client -connect mysite.dev:443 2>/dev/null \ | openssl x509 -noout -enddate | cut -d= -f2)" +%s) - $(date +%s)) / 86400 )) [ "$days" -lt 15 ] && alert "certificate expires in ${days} days"

When the project grows to the point of needing history and graphs, put the observation stack on another machine. It should never share memory with what it observes.

What actually knocks on your door

This section is usually abstract, so I went and got real numbers. This is the SSH journal of an ordinary Ubuntu server, nothing special about it, no famous name, exposed to the internet:

window: 2026-07-20 to 2026-09-20 (62 days) authentication failure lines: 112,165 of those, "Invalid user": 53,813 distinct IPs: 2,834 average: ~1,810 attempts a day, ~75 an hour, without pause

Seventy-five attempts an hour, non stop, on a machine nobody has heard of. There is no such thing as "my project is too small to be a target", because there is no target: there is a scan of the entire internet, all the time.

And the usernames tried tell a story:

8,012 admin 4,493 user 3,998 solana 2,701 solv 2,505 debian 2,063 sol 1,148 test 685 oracle 564 ftpuser 546 support

The three highlighted are the same target. They are bots hunting for Solana validator nodes, which tend to be machines holding a private key and a balance. Those numbers say what automated attacks are hunting for this month, and the list shifts with whatever is currently worth money.

fail2ban banned 209 IPs over that window. And it is worth being honest about what that means: 209 out of 2,834 is 7%. fail2ban is hygiene, it cuts noise in the log and costs little. It is not your defence. Your defence is part 1's three lines: key-only authentication, no root login, and a firewall that denies by default. With those, all 112 thousand attempts are irrelevant, because no password exists to be guessed.

What holds up over time is short:

apt list --upgradable # unattended-upgrades handles the security ones ss -tulpn # what is exposed? should it be? systemctl list-timers # is certbot still alive? journalctl -p err --since "7 days ago"

That ss -tulpn is the highest return one. The question is always the same: does every process listening on 0.0.0.0 really need to be reachable from the world? Almost always the answer is no, and almost always it is a database or a Redis that came up with the distribution's default.

When 2 GB stops being enough

The whole series argued that far more fits on a small box than the service catalogue lets you imagine. Closing without saying where the limit is would be dishonest.

The signals, in the order they show up:

1. Swap stops sitting still. By part 1, occupied swap is healthy; si and so hammering in vmstat is genuinely running out of RAM.
2. available does not come back. After a spike it should return to its baseline. If it does not, something is growing.
3. The database becomes the bottleneck. Connections queueing, queries waiting on work_mem, Postgres cache too small for the active dataset.
4. Deploys get risky. When there is no room for part 4's 438 MB peak even with a ceiling, the machine is full.

And the right order of next steps, which is almost never the one the hype suggests:

1. Double the machine. 4 GB costs roughly twice 2 GB and fixes it with no architectural change at all. It is the cheapest option once you count your own time, and it is the one people skip because it does not feel like a solution.
2. Move the database to its own machine. The first split worth making, because the database has the most different memory profile of everything there.
3. Move static files and images to object storage with a CDN. Cheap, and it gives back disk and bandwidth.
4. Only then think about more than one application server, a load balancer, shared sessions, and all the operational cost that comes with them.

Container orchestration is not on that list, and the reason is part 1's budget. A Kubernetes control plane consumes, on its own, more memory than the entire stack this series assembled. It solves real coordination problems across many machines. None of those problems are yours yet.

The checklist for the whole series

Five parts on one page. If all of these are true, you have a server you understand:

Twenty command lines, five configuration files and one deploy script. That was it. No dashboard, no YAML, no piece you cannot open and read.

What these five parts really deliver is not a configured VPS. It is the ability to answer what is happening on this machine right now, which is the question the managed platform had been answering for you, in exchange for you never having to learn. When you go back to the platform after this, and you will, because for plenty of things it is the right call, you start choosing it for a reason instead of for lack of an alternative.

Two gigabytes go a lot further than you were told.

kelvin@cluxnei:~/blog$