I Audited My Own Production Server: Ten Faults, Every One of Them Reported Success

A deploy printed Deployment complete!.

Every downstream check agreed with it: all containers healthy, /up returning 200, the home page and list pages rendering, migrations applied without error. The whole pipeline green.

It had deployed the previous release.

---

This is a first-hand account of a two-day end-to-end audit of SideShip's own production server. The host is a GCE e2-medium with a 9.6G root filesystem running Postgres, Redis, a Rails app, nginx and certbot - an entirely ordinary side project deployment. The audit found ten real faults, all now fixed, and every number in this article traces back to the git history, a code comment, or output I measured on that machine myself. Where I could not measure something, I left it out, and in several places below I say explicitly "this is an inference, not a measurement."

Not one of these was "the site is down." What they have in common is something harder to deal with:

Every one of them produced a success signal.

The deploy printed "complete". Pages returned 200. The log size cap was confirmed present on all five containers. A backup runbook sat in the repo, reading exactly like a configured system. The CSS built with no warnings. docker logs was quiet.

The defect is not missing monitoring. The defect is monitoring that answers a question adjacent to the one you meant to ask. Status codes answer "is it up", not "is it right". systemctl list-timers answers "is it scheduled", not "did it work". docker inspect answers "is the cap configured", not "does the cap govern this stream". Exit code 0 answers "did the command run", not "did anything change".

Three parts: the ten faults (with the real numbers and why nothing noticed), the shape they share, and a fourteen-item checklist you can run tonight. Every checklist item is labelled with whether I actually ran it and what it printed.

A note on the boundary of this article. This is a live machine. I describe the shape of each defence and the reasoning behind it, but never the parameters - rate-limit thresholds and windows, spam-detection signatures, the honeypot field name, credential file paths, accounts and IPs are all deliberately absent. What you should take away is what to check and why, not how to get past ours.

---

1. The deploy printed "complete" and ran the previous release

Start with the one at the top.

make deploy was a sequence: git pull, build, up, migrate, print Deployment complete!.

Run as root, git pull fails - root can read the checkout but does not hold the GitHub deploy key. Nothing checked. Every following step then ran happily against the unchanged working tree: the image was rebuilt from the old code, containers were recreated, migrations ran, and the deploy reported success.

The identical failure recurred the same day through a different door. After moving builds to CI, deploying against the :latest tag has exactly the same shape: CI publishes :latest and :<sha> together, but minutes after the push, and the workflow's concurrency group cancels a superseded run when two commits land close together. In that window git advances the checkout, docker compose pull fetches the previous commit's image, up restarts on it, migrate runs, and it prints Deployment complete! while serving old code.

Why nothing noticed

Because the deploy printed Deployment complete!. That is the whole finding.

Every downstream check agreed: containers healthy, /up 200, the site serving pages, migrations applied. Health checks answer "is something running", never "is the right thing running." The version actually being served was never compared against the version requested.

The fix

Pin the deployed image to the commit that is actually checked out: sha=$(git rev-parse HEAD), then pull :<sha>, never :latest. If CI has not published that SHA yet, the pull exits non-zero and make aborts before anything is touched.

The line from the commit message is worth lifting out on its own:

"Not deployed yet" is a fine outcome. "Deployed something other than what you asked for" is not.

The supporting changes each encode a specific trap:

• docker compose pull app is its own step, so a pull failure stops the deploy there rather than being carried past.
• up carries --no-build. The app service declares both image: and build:, and plain docker compose up will silently build a missing image - verified against the daemon, not assumed. That turns a routine restart into a large unguarded build.
• The advice to pin an image tag in .env was removed. Compose auto-loads .env, so a pinned tag makes every future deploy resolve to the same old image, succeed, and report success. That is the same fault wearing a more insidious costume.
• The pull failure message now branches on Docker's own error text - manifest unknown means CI has not published yet, unauthorized means credentials - instead of printing a vague guess.

That unauthorized is not hypothetical; it happened on the first real registry deploy. Docker registry credentials are per-UNIX-user, stored in that user's own home directory, and are not shared via the docker group - so a sudo docker login does nothing for the account that actually runs make.

The generalisation is the useful part: a pipeline of steps in which only the first can fail meaningfully, and the rest succeed regardless of what happened before them, is a pipeline whose success message means nothing. Each step's success criterion was "this command exited 0", not "the system now runs the artefact I asked for".

This one is qualitative. The repo records that it happened, but records no count and no duration, so I am not attaching a number to it.

---

2. Every page returned 200 and every user-uploaded image was broken

The commit that moved the build to CI also did something else: it silently dropped a gitignored credential file.

This site's images live in cloud object storage, and Active Storage authenticates with a service-account key. That key has never been in version control - it is gitignored - but a copy sat in the server's own checkout, placed there by hand eight months earlier.

And the Dockerfile contains COPY . ..

So while the server built its own image, COPY . . picked the key up out of that checkout and baked it in. Nothing declared this. It was a side effect. When the build moved to GitHub Actions, CI checked out from GitHub, where the file has never existed, so the published image simply did not contain it.

The environment variable was still set, so production still selected cloud storage, and the blob redirect raised at the authentication call.

Every user-uploaded image on the site returned 500. Every page containing them returned 200.

The exposure window

10 hours 15 minutes: from the CI-build switch (2026-08-04 13:29:36 +0800) to the fix (23:45:05 the same day).

That window is inferred from commit timestamps, not measured. Git does not record when a human first noticed. The commit message says only that the images were all broken that day and the author did not notice. I am not going to invent a discovery story - **what the record establishes is that nothing detected it.

Why nothing noticed

This is the best illustration on the list, because every layer of verification was structurally blind to it, and each for a different reason:

1. Post-deploy verification checked HTTP status codes on /, /projects and /blogs. All three returned 200 - because what broke was the <img> inside the page, not the page.
2. RSpec cannot catch it: the test suite never talks to cloud object storage.
3. The CI image assertions cannot catch it: they verify the runtime user, file permissions and directory writability, not whether an untracked file CI has never seen is present.
4. The uptime endpoint returned 200 throughout.
5. The image URLs are Active Storage redirects, so a naive HEAD on the page's HTML never reaches the object at all.

The failure lived in the one gap between "the page loads" and "the page is correct", and no status-code check can see into that gap.

The fix

The fix is itself the second lesson. The key was not put back into the image; it is bind-mounted read-only from the server's checkout into the container. The commit argues the reason explicitly: the image is now pushed to a registry, so anything baked into it is extractable by anyone who can pull it, and stays in the layer history forever even if a later commit removes it. Credentials belong to the host, not to the artefact.

Then the missing piece: a check called make smoke, wired into the end of make deploy. It fetches a real list page over the public URL, extracts real Active Storage URLs from the HTML, and follows the redirects all the way to the objects - the first five it finds.

And it was proven to fail before it was trusted. The credentials file on the server was replaced with {} and the app restarted: smoke reported 5/5 failed, exited non-zero, and pointed straight at the file to check. The file was restored: 5/5 passing.

The commit message contains the best single line in the whole audit:

A check never proven to fail is not a check.

Generalised: COPY . . makes your image a function of one particular filesystem, not of your repository. You find out which files were load-bearing on the day you move the build.

---

3. make clean said it removed images; it also removed the production database

This one was found by reading code, not by anyone triggering it. The database was never actually lost.

The target was:

clean:
    docker compose down --rmi local --volumes --remove-orphans

--volumes removes the named volumes declared in docker-compose.yml. On this host that is three: the production database, uploaded files, and the TLS certificates.

make help described the target as "Remove stopped containers and images".

The word "volumes" appeared nowhere in the help text.

Why monitoring cannot catch this

By definition it cannot. This is not a malfunction, it is a loaded gun. Nothing is wrong until someone runs the command, and then everything is wrong at once and irreversibly.

It also passes every automated check: the Makefile is syntactically valid, CI is green, the target does exactly what it says. The only detector is a human reading the recipe and comparing it against the help text.

The Makefile comment spells out the realistic failure path, and that is what makes it worth publishing:

A deploy runs out of disk mid-build, the operator scans make help for something that frees space, sees "clean", and deletes production.

The command an operator reaches for under pressure must be the safe one.

The fix

clean was rewritten to prune stopped containers, dangling images and build cache only - it now cannot touch a volume.

Deliberately not docker image prune -a: -a would delete the current app image whenever the stack happens to be down, forcing a full rebuild on a host that cannot afford one.

The destructive behaviour moved to a new destroy-all-data target that requires typing DESTROY at a prompt; non-interactive callers get an empty read and abort. make help gained an explicit "Danger zone" section, and the safe target's help now states positively that it touches no volumes and that the database, uploads and TLS certificates are safe.

A related guard in the same family: the pull preflight threshold was deliberately set lower than the build threshold, on the reasoning that a guard which blocks safe operations gets worked around - and being blocked while hunting for disk space is precisely the situation in which make clean nearly destroyed the database.

---

4. The log size cap was configured on all five containers and governed nothing

On 2026-07-28 the disk was already tight (85%, 1.5G free of 9.6G, with the nginx access log alone at 109MB), so the right thing was done: a json-file driver cap of max-size 10m / max-file 3 was applied to all five services (db, redis, app, nginx, certbot) via a YAML anchor in docker-compose.yml, plus the same default on the host daemon for future containers.

docker inspect confirmed it: all five containers carried the limit. Verification passed.

The app container ignored it completely.

Because config/puma.rb contained this line - inherited from Rails' default puma.rb template, predating containerisation entirely:

stdout_redirect '/…/puma.stdout.log', '/…/puma.stderr.log', true if rails_env == 'production'

Puma reopens its own stdout/stderr onto files inside the container, at a path mounted on a named volume.

Docker's log driver only governs what the daemon receives on the container's stdout/stderr. A process that writes files inside the container never reaches the driver. The cap governed an empty stream.

The files grew without bound. And docker logs app returned nothing at all.

The numbers

• 965MB of puma logs accumulated in the volume
• Disk at 87% (root filesystem 9.6G total)
• docker logs: 0 lines over 7 days

Why nothing noticed

This is the sharpest example on the list of a fix that hid its own failure.

Adding the cap felt like fixing the problem, and the verification step passed - docker inspect confirmed all five containers carried the limit. The one container that mattered was exempt for a reason invisible to that check, because the exemption lived in application code, not container config.

Worse, the bypass destroyed the very instrument you would use to notice. With docker logs empty, the normal "is anything weird in the logs" habit returns nothing - and empty reads as "quiet", not as "broken".

The code comment records the price directly: it "left the 2026-07-31 503 outage undiagnosable".

The fix

Remove the line, and replace it with a comment explaining why it must never come back: production already logs to $stdout; the cap is in docker-compose.yml; and if you ever need Puma's own output in a file, do it on the host (docker compose logs app > file), never with stdout_redirect.

Rails output returned to stdout, the existing 10m x 3 cap took effect immediately, and observability was restored as a side effect. The 965MB already in the volume was dead weight and was deleted once by hand. The log mount was deliberately kept, so any stray in-container write lands somewhere inspectable and prunable rather than in the container's writable layer.

The root cause in one sentence: two independent log-rotation mechanisms were assumed to be one. The container-level cap was the newer, deliberate control; the stdout_redirect line was inherited and older. Redirecting stdout to a file is correct for a bare-metal Puma and is exactly wrong inside a container, where stdout is the log transport. And nothing warns you: the redirect succeeds, the cap is applied, both look configured.

---

5. 89,135 stack traces, all from bots probing for .php files

config/routes.rb ends with a catch-all:

match '*path', to: 'errors#not_found', via: :all,
               constraints: ->(req) { !req.path.start_with?('/rails/') }

And ErrorsController#not_found used a respond_to block with only format.html and format.json.

Bots probing for /1.php, /admin.php, /wp-content/plugins/…/wp_filemanager.php and so on request a format Rails cannot resolve to a registered Mime::Type, so request.formats came back empty and request.format was a Mime::NullType.

respond_to with no matching clause raises. It does not fall back. The exception is ActionController::UnknownFormat, which in production surfaces as a 500 - not the intended 404 - and Rails logs a full backtrace for every single one.

The same bug also hit legitimate crawlers: the real route is get 'sitemap', so Bingbot's request for /sitemap_index.xml fell into the catch-all too and received a 500.

The numbers

89,135 occurrences of ActionController::UnknownFormat in the production log.

That figure has an independent corroboration. One week of nginx access logs, measured the same day:

200 is the second-smallest number in that table. The 4,232 weekly 500s corroborate the storm independently of the 89,135 cumulative figure. For background, this host takes roughly 1,300 scanner-bot requests a day.

Ten hours later, the same shape came back through another door

While checking whether the newly deployed rate limiting had caught any real users, the variant surfaced: the catch-all also answers POST, and Rails enables forgery protection on every controller by default, so scanner POSTs to nonexistent paths (POST /dns-query, POST /vpnsvc/connect.cgi) raised ActionController::InvalidAuthenticityToken - a 422 plus a full backtrace each time.

The post-fix volume shows how routine this shape is: 6 occurrences in 6 hours from 4 scanning IPs. The volume is small, but it scales with whoever is scanning you, not with anything you control.

The fix was skip_forgery_protection on the errors controller. That is safe here in a way it would not be elsewhere: these three actions render a static error page and change no state, so there is nothing to forge a change to. The honest answer to POST /dns-query is 404 - the path does not exist, and no CSRF token would have made it exist.

Why nothing noticed

Three compounding reasons.

(a) Nobody requests /admin.php by hand. The failing path is one no human ever exercises - it is reachable only by traffic you are not watching.

(b) The visible symptom was a 500 to a bot, and nothing was configured to alert on 500s.

(c) Critically, the cost was not the errors, it was the bytes. Each occurrence writes a full backtrace, and 89,135 of them went into a log file that had no size cap and that docker logs could not show (see the previous section). So the one metric that would have revealed it - log volume - was the one thing rendered unreadable.

Test suites cannot catch it either: RSpec never sends a request with an unregistered extension, and the errors controller had no request spec at all until this fix.

The fix

One shared private method for all three actions, and the clause that matters is format.any:

def respond_with_error(status, message)
  respond_to do |format|
    format.html { render status_page(status), status: status, layout: 'application' }
    format.json { render json: { error: message }, status: status }
    format.any  { head status }      # must stay last
  end
end

format.any must stay last so html and json keep winning content negotiation. The root cause in one line: respond_to is a whitelist, and an unmatched format is an exception, not a fallback. A controller whose entire job is answering requests for paths that do not exist was written as if it would only ever be asked for html or json.

The same commit fixed two more things: :unprocessable_entity became :unprocessable_content (Rack 3.2.4 no longer maps the old symbol, so /422 raised ArgumentError), and -

a second live 500: Project.search queried a description column that does not exist (description is ActionText), so any search on /projects returned 500. That scope had zero test coverage, which the commit message states as exactly why it shipped.

Those two bugs together are instructive because they are opposite lessons: one was structurally unreachable by tests (nobody writes a request spec with a .php extension), the other was simply untested. The fix also added the errors controller's first request spec: 29 examples, since grown to 31.

---

6. Errno::ENOSPC: disk is the slow variable

The previous two sections were feeding the same thing.

In March 2026 this box hit Errno::ENOSPC 127 times and took the app down.

That number appears in three places in the repo, but all three are the same author recalling it - there is no log artefact. **Month precision, no exact date, and I am not adding precision it does not have.* It was counted at the time; it cannot be recounted now.*

Separately, there was a 503 outage on 2026-07-31 that could not be investigated at all, because the app's logs were going to a file docker logs could not read.

I am keeping those two apart deliberately. No source states they were the same event, and the log-cap work sits between them. Note also the ordering: the catch-all bug in section 5 was not fixed until 2026-08-04, so the 89,135 backtraces were still accumulating through the July outage.

Where the bytes came from

The link "89,135 backtraces → the 965MB volume → ENOSPC" is a consistent inference, not a single sourced fact.

One commit says the 89,135 UnknownFormat traces were the main cause of the disk being filled; a different commit independently says the puma log volume reached 965MB. Rails backtraces go to the app log, and the app log was being redirected into that volume, so they are almost certainly the same bytes - and the 89,135 must have been counted from that file, since docker logs was empty.

But no single source says so. So I am writing it as reasoning, not as a measurement.

The real shape of the disk (measured 2026-08-04)

That build needs about 87% of the available headroom - the repo's own figure, and it is the number that matters, not the tidy arithmetic. And during a single day of deploys on 2026-08-04, the disk hit 93% twice.

Which is why "disk percentage" is the wrong threshold. The right threshold is whether a deploy's transient requirement still fits.

Why nothing noticed

There was no disk monitoring of any kind - no alerting, no dashboard.

Disk is the classic slow variable: it degrades monotonically over months with zero symptoms, then fails all at once and takes everything with it, including the mechanisms you would use to diagnose it.

There is also a second-order failure mode worth knowing about: a filling disk does not present as "disk full". On another host I run, it presented as Redis being unable to write its RDB, entering MISCONF, and refusing all writes - which surfaces as site-wide 500s. A symptom that points away from the cause. That one is recollection, not something this repo records, so I am not attaching a duration to it.

And ENOSPC arrives during a write, so the first thing that breaks is often the log that would have told you.

The fix, layered

1. json-file cap of 10m x 3 on all containers, plus the same default on the host daemon.
2. Remove the stdout_redirect that was bypassing it (section 4).
3. make preflight refuses to start a build below a free-space floor. One implementation detail is worth copying: measure against the filesystem docker info reports as DockerRootDir, not /. On a developer's macOS, / is the sealed read-only system volume and always reports about 2G, which would block every local build for no reason.
4. make build runs docker builder prune -f immediately after (reclaimed 841MB when measured); make deploy prunes and prints df; make reclaim exists as a standalone volume-safe target.
5. Finally, image builds moved off the host entirely to GitHub Actions plus a registry, so the server only pulls. A reclaim-app-images target was added because docker image prune only removes dangling images, and every rollback-pulled :<sha> is tagged - leaking ~914MB per rollback, forever.

---

7. No database backups had ever existed - and cron was not even installed

The production database had never been backed up, at all, since the app went live.

BACKUP.md in the repo contained a section headed "Scheduled Backups (Cron)", instructing the reader to run crontab -e and add a nightly make backup.

That procedure had never been carried out, and could not have been: cron is not installed on this Ubuntu host.

The audit checked the server directly rather than trusting the documentation: no crontab, systemctl list-timers showing no backup timer, and a filesystem-wide search for .sql / .sql.gz returning nothing.

For how long

The earliest deployment documentation commit is 2025-12-13, and the cloud-storage service-account key was placed on the server by hand the same day, so the host has been serving since approximately mid-December 2025. Backups were implemented 2026-08-04.

That is roughly 7.5 months - about 234 days - with a production database and no recoverable copy of it.

Treat the start date as "approximately mid-December 2025". It is inferred from commit dates, not from a server record.

Why nothing noticed

Backups are the canonical zero-signal system: a working one and a nonexistent one produce identical observable behaviour every single day, and they differ exactly once - on the day you need one. There is no page that 500s, no metric that moves.

Worse, the two things that feel like verification both lie here:

• The runbook said backups were scheduled. Documentation was mistaken for implementation. A written procedure in a repo file reads exactly like a configured system, and nothing in the deploy pipeline ever asserted otherwise. Compounding it, the procedure was impossible on this OS image, so even a diligent operator following the doc would have gotten an error and had to improvise.
• systemctl list-timers - had a timer existed - reports a healthy next-run time regardless of whether every previous run failed.

The only honest check is "when did a dump last succeed, and can it be restored".

The fix

A backup script plus a systemd service and timer. Each design point encodes a specific failure:

• Dumps are written to an .inprogress temp file and only renamed into place after they are verified non-empty, contain pg_dump's completion marker, and pass gzip -t. Any failure exits non-zero.
• Backups live outside both the git checkout and the Docker build context, because the Dockerfile does COPY . . and Docker does not read .gitignore - nightly dumps inside the repo would be baked into every image layer. (A .dockerignore entry was added as defence in depth.)
• Retention measures and deletes only the project's own *.sql.gz files, and never deletes the newest. An earlier draft measured the whole directory with du -sm but deleted only dumps - so dropping one unrelated tarball in there would have deleted every database backup and reported success.
• When space allows it dumps first, then prunes (a failed run should never reduce the backups you already hold). Only when space is short does it prune first, because at that point the choice is between having a backup and not having one.
• It writes a last_success status file so --status can answer "did last night's backup succeed?" without docker and without the database.

The old make backup was replaced too. Its inline pg_dump checked no exit code, so running it with the db container down produced a ~20-byte file named exactly like a real backup - which then sat in the retention queue ahead of genuine dumps.

That 20-byte fake is the direct parent of checklist items 7 and 8 below: a file existing is not a backup existing, and a backup existing is not a backup containing data.

---

8. 1,778 spam rows - and my first conclusion about them was wrong

POST /feedbacks required no authentication, no captcha, no rate limit and no honeypot. A scanner found it and filled the table.

1,778 rows.

For scale, here is the same production snapshot taken on 2026-08-04: 7 users, 5 published projects, 0 comments, 1 praise, 2 conversations, 12 messages.

The single most-used write endpoint on the site was the one being used by a bot.

All 1,778 rows were created on 2026-04-23; zero on any other day; zero belonged to a signed-in user. From that day to the audit is three months and twelve days.

And then the half I got wrong

The audit's first conclusion, written into a commit message, was that all 1,778 rows were one repeated payload - a single email/content pair.

That conclusion came from eyeballing the first five rows.

When the purge rake task was actually dry-run against production the next day, the narrow predicate matched only 705 of 1,778. The other 1,073 would have been silently left behind by a cleanup that reported success.

Those 1,073 rows were SQL-injection scanner payloads of varying shape - sqlmap-style probes: bare digit strings, boolean-arithmetic tautologies, time-delay probes using sleep/waitfor, and an Oracle DBMS_PIPE.RECEIVE_MESSAGE probe. They were not spam content in the normal sense. They were a scanner walking the field looking for an injectable parameter.

The deeper root cause is one sentence: the assumption "a burst from one scanner means one payload" is wrong, because scanners vary their payload by design. That is what a scan is.

The wrong conclusion still survives in four files in this repo - a model, a service, an initializer and a deployment doc - because they were written before it was checked and were never updated after the correction. The right figure lives in the rake task's comment and in the correcting commit. I am leaving that in the article rather than quietly tidying it, because it is more instructive than a clean narrative: **a finding written down before it was verified spread to four files.**

Why nothing noticed

Nothing counted anything.

No admin view listed feedback. No metric tracked row growth. No log line recorded a submission.

Four writes a minute for one afternoon looks identical to zero writes if nobody is looking - and on a site with 7 users, nobody was.

The second-order lesson is the more useful one: the correction was also nearly invisible. A cleanup script that matches a subset exits 0 and reports success. Had the task not printed matched-vs-total in its dry run, 1,073 rows would have survived a purge that claimed to have worked.

The fix

Shapes only, no parameters:

• Per-IP and per-identity limits on the abusable write endpoints, plus a global ceiling tuned so that one ordinary page view - which fans out into far more requests than people expect - can never 429 a real visitor. Which request classes that ceiling counts, and where any of the thresholds sit, stay out of this article.
• A hidden decoy form field plus a signed, expiring render timestamp.
• An environment-variable kill switch, so a misfiring limit is a ~10-second .env change rather than a code-CI-deploy cycle.
• Every match writes one line to the same stdout the app uses, so docker compose logs app | grep rack-attack answers "is someone hammering us right now" without database access. That directly fixes the "nothing counted anything" part.

The cleanup task was deliberately gated. It has a whole-day mode, but it aborts if any row that day belongs to a signed-in user, aborts on an unparseable date, prints how many rows on other days it is leaving untouched, and refuses above a maximum row count - on the reasoning that if the predicate matches more than that, the predicate is wrong, not the data. All three abort paths were tested against a test database.

---

9. A 1.39MB stylesheet, two thirds of which was a map of the file to itself

The compiled application.css shipped to every visitor was 1,393,531 bytes.

917,917 of those bytes - 65.9% - were not CSS. They were a base64 inline sourcemap appended by postcss-cli.

The build chain runs sass (with --no-source-map, correctly) writing application.css, then postcss reading and writing that same file in place to run autoprefixer.

postcss was told nothing about maps, and it generates one by default. Because it read and wrote the same path, the map it produced had a sources entry pointing at application.css - the file itself.

A map from the file to the file. About 918KB of it, on the render-blocking path, downloaded by every visitor before first paint.

The numbers

I re-measured both post-fix figures in the working tree while writing this, and they match the commit message exactly: wc -c = 472,411, gzip -c | wc -c = 61,844, grep -c sourceMappingURL = 0.

Note the arithmetic: 1,393,531 − 472,411 = 921,120, which is the 917,917-byte map plus about 3.2KB of genuinely dead SCSS removed in the same commit (leftover blocks from features that had been taken out). The CSS content itself did not change by a single byte.

Why nothing noticed

Nothing about it is an error. The build succeeds, the CSS is valid and complete, every page renders correctly, tests pass, no console warning appears.

And browsers do not fetch or parse an inline sourcemap unless DevTools is open - so it costs bandwidth and first-paint latency while producing no visible symptom at all.

The only detector is looking at the size of the artefact you actually ship. On a site with a handful of visitors, no page-speed alert would ever have fired.

The fix

Add --no-map to the postcss step. One flag.

Deliberately not done: Bootstrap tree-shaking. The stated reasoning is that it is the genuinely risky change - broken styles raise no error, they just quietly look wrong, and on a low-traffic site that could go unnoticed for weeks.

The generalised checklist item is cheap: after any build-tool change, run ls -l on your built assets and ask whether the number is plausible. A stylesheet that is megabytes is telling you something even when nothing is broken.

---

10. Social preview images broken two independent ways at once

Two separate faults, either of which alone would have killed every social preview on the site.

Fault one: the project and user pages set og:image and twitter:image via url_for(@project.banner.first).

url_for on an Active Storage attachment returns a path, not a URL. Open Graph requires an absolute URL, so Facebook, X, LinkedIn, Slack and Discord all rendered no preview - made worse by twitter:card being summary_large_image, a layout that is almost entirely image and therefore looks worse when empty than the plain card would.

Fault two: the layout's fallback for every page without its own image pointed at /og-image.png, and that file did not exist. Confirmed 404 in production, and absent from both version control and the Dockerfile.

So the home page, the blog, and any project without a banner - most of the shareable surface - had no preview image at all, and never had.

Why nothing noticed

Both faults are outside the request/response cycle the site itself ever exercises.

The pages render perfectly, return 200, and pass every test - the tags are present in the HTML, and no test asserted that their values resolve to anything.

The damage happens on someone else's server, when Facebook or Slack fetches the URL, and it is reported to nobody: no error, no log line, no status code on your side.

For a site whose growth model is people sharing project links, this was a failure sitting directly on the primary growth path, invisible by construction.

The fix

Both pages switched to rails_blob_url, which takes host and scheme from the current request and therefore cannot emit localhost even though the app sets no default_url_options.

A defensive normalisation was added in the layout: whatever any page passes through content_for :og_image is coerced to an absolute URL there, so the next person to add one cannot repeat the mistake. Plus og:image:secure_url when the result is https, and a guard so calling rails_blob_url on an unattached avatar cannot raise.

And the missing fallback card was generated in the design system's brand green. Verified in the working tree today: 1200 x 630, 8-bit RGB, non-interlaced PNG, 76,823 bytes.

This incident is qualitatively real but has few numbers. I am not manufacturing engagement or click figures for it.

Three more SEO faults in the same family, same commit

• The blog sitemap's lastmod was File.mtime, which inside a Docker image is the build time - so every deploy told crawlers that every article had just been updated.
• /projects had no title or description of its own, byte-for-byte identical to the home page.
• The zh-TW and en article pairs carried no reciprocal hreflang, so search engines treated them as duplicates.

And three faults the fix itself introduced, caught by testing rather than shipped

I find this trio more valuable than the three above:

1. Adding a frontmatter date 500'd every article, because the blog controller's YAML.safe_load lacked permitted_classes: [Date] - while the sitemaps controller had had it all along.
2. The new hreflang="en" pointed at a URL that returned <html lang="zh-TW">, because the blog controller never set I18n.locale from the article's language. A contradictory hreflang is worse than none.
3. An existing sitemap spec asserting "an article with no date must not get a fabricated lastmod" became a false pass the moment every article had a date. It was rewritten to build its own fixture.

That third one is my favourite small thing in the whole audit: a test can stop testing anything without being modified at all.

---

11. Not in the top ten, but equally real

These came out of the same audit. Several are better checklist items than some of the ten.

• Project.search queried a column that does not exist, so any search on /projects returned 500. It had zero test coverage, which the commit message states as the reason it shipped. Same commit and same day as the catch-all bug in section 5, but the opposite lesson: one was structurally unreachable by tests, the other was simply untested.

• No CI existed at all until 2026-08-04. The price already paid, stated in the commit: 153 RuboCop offences unnoticed, the test suite hanging 3 runs out of 4 with nobody aware, and a broken scope reaching production. Also: CI runs RAILS_ENV=test throughout, so a typo in production.rb is invisible until deploy - a separate CI step now boots the production environment, because this repo had already paid that cost once.

• An intermittent 50% test-suite failure rate traced to the layout loading Bootstrap Icons CSS from a CDN, so headless Chrome made a real network call during system specs. The symptom presented as "cannot find button X" - it looked like an application bug and sent the investigation the wrong way. It was also a production dependency on a third party being reachable and fast, with the CDN pinned to a different version than package.json.

• Devise :recoverable was enabled with no SMTP configured anywhere. The site offered "forgot password", accepted the form, and did nothing. With 7 users, one person forgetting a password is 14% of the userbase permanently locked out. The fix was to stop advertising a door that does not open.

• A chown -R placed after COPY in the Dockerfile created a 32.7MB duplicate layer on every commit, because chown rewrites inode metadata and overlayfs cannot record that without copying every touched file up a layer. Measured by building both revisions from an identical clean checkout: 601,095,332 → 568,386,052 bytes, that layer 32.7MB → 8.9kB, and the COPY layer byte-identical. The simpler-looking COPY --chown was rejected because it would make all application code writable by the runtime user.

• A dead-code removal that doubles as a "how do you know it's dead" case study. Establishing that Tag/Tagging was dead required excluding image_tag, stylesheet_link_tag, csrf_meta_tags, TagBuilder, a bi bi-tag icon class and a tagline i18n key - every single hit was a false positive. Reversibility was proven by rolling the migration back and diffing the regenerated schema.rb byte-for-byte against the original.

• An inline CKEditor 5 init script sat in two comment views while CKEditor appeared in no package.json, yarn.lock, importmap, Gemfile or vendor directory. It never exploded only because it was guarded by typeof CKEditor5 !== 'undefined'. 23 lines deleted.

• A GNU make trap worth a checklist line: any recipe line textually containing $(MAKE) executes even under make -n, so a multi-command shell block containing it turns a dry run into a real deploy. Verified in this repo, and worked around by aliasing make to a differently-named variable.

---

12. The shape they share

Put the ten side by side and a structure appears.

None of them was caught by checking whether the site was up. Each produced a success signal at its own layer:

In one line: the defect is not missing monitoring, it is monitoring that answers an adjacent question.

• Status codes answer "is it up", not "is it right".
• systemctl list-timers answers "is it scheduled", not "did it work".
• docker inspect answers "is the cap configured", not "does the cap govern this stream".
• Exit code 0 answers "did the command run", not "did anything change".
• An empty docker logs answers "nothing reached the driver", but reads as "nothing happened".

Which is why every item in the checklist below is phrased as the question your usual check does not answer.

The second thread: checks proven to fail

Two of the fixes were validated by being proven to fail first:

1. The smoke test was run against deliberately broken credentials: 5/5 fail, then restored, 5/5 pass.
2. The CI image assertions were run against an alpine image to confirm they failed 7/7 rather than false-passing. The first version of that assertion did false-pass - because touch failed on a nonexistent path rather than on a permission.

The repo's own phrasing is the best version of it:

A check never proven to fail is not a check.

And it happened to me again while writing the checklist below. My first version of the image check tested Content-Type, and my own fixture caught me out: a static server derives Content-Type from the file extension, so a .png full of HTML still reports image/png and the check false-passes. You have to read the actual bytes - the magic number.

That is this article's own thesis happening to the check itself.

---

13. Fourteen checks you can run tonight

Each item is labelled with whether I actually ran it and what it printed.

Three Docker-related items were not executed locally (items 3, 4 and 11) - the owner of this machine asked for Docker to stay off. For those, I checked every flag and subcommand against the official docs, ran the surrounding shell through a sh -n POSIX syntax check, and exercised the classification logic against stand-in programs that produce docker's output structure. The label is not run locally (shape-checked against docs) - which is more honest than presenting a command nobody ran as tested.

The other eleven were actually run, and most were verified in both the good and the bad direction, exit codes included.

Portability notes: du's -x and -d work on both BSD (macOS) and GNU (Linux); --max-depth is GNU-only, don't use it. systemctl is Linux-only. macOS's df / severely under-reports free space because / is a read-only system snapshot volume, so using it as a threshold there will mislead you.

---

1. Is the disk nearly full?

df -h /

• Bad: utilisation near capacity. But note - this command only answers "how much is left", never "who ate it", so it is always just step one.
• Good: clear headroom. And the real threshold is not a percentage, it is whether a deploy's transient requirement still fits - measure that number first, then come back and ask whether this percentage is enough.

Actually run. Local output: 86% used, 2.3Gi free. It also demonstrated the macOS trap above: on macOS / is a read-only system snapshot volume and this number under-reports badly. On a Linux host the two are the same thing.

---

2. What is actually eating the disk?

du -x -m -d 1 / 2>/dev/null | sort -rn | head -12

• Bad: some directory you cannot explain sitting near the top. Especially /var - if it is large and you don't know why, this command has taken you as far as it can and you need to go into container storage (next item).
• Good: the largest directories are ones you recognise and can account for. Units are MB, largest first.

Actually run, in two configurations: the full du -x -m -d 1 / (about 7 minutes on this 228G Mac, but seconds on the kind of 10G host this article is about) and bounded subtrees.

---

3. Is the biggest consumer hiding inside a Docker volume?

docker system df -v

• Bad: some volume you assumed was irrelevant - a log volume, say - sitting at the top. This is exactly the blind spot host-side du cannot see into: du tells you /var is large, not which volume, and certainly not why.
• Good: the Local Volumes section at the bottom lists every volume by name and size, and the largest one is the one you expect (the database, presumably).

Not run locally (shape-checked against docs). Confirmed that -v/--verbose emits a Local Volumes section with per-volume name, link count and size. The docker-free equivalent (Linux, needs root) is du -sm /var/lib/docker/volumes/* | sort -rn | head, whose output structure I verified by running it against a substitute directory.

---

4. Is there a log cap - and does anything actually reach the driver?

This item is the direct product of section 4.

for c in $(docker ps --format '{{.Names}}'); do
  printf '%-20s %-10s %-8s %s lines/24h\n' "$c" \
    "$(docker inspect -f '{{.HostConfig.LogConfig.Type}}' $c)" \
    "$(docker inspect -f '{{index .HostConfig.LogConfig.Config "max-size"}}' $c)" \
    "$(docker logs --since 24h $c 2>&1 | wc -l)"
done

• Bad - catch both:
  1. The max-size field is empty. The json-file driver does not rotate by default, so this container can fill the disk on its own.
  2. The insidious one: max-size looks perfectly correct, but 24 hours shows 0 lines for a service that is demonstrably serving traffic. The cap isn't wrong - nothing is reaching the driver at all because the application is writing files itself. The cap governs an empty stream.
• Good: every container has a max-size value, and no line count is 0.

Not run locally (shape-checked against docs). Confirmed: docker inspect -f '{{.HostConfig.LogConfig.Type}}' is the documented form; json-file's max-size defaults to -1 (unlimited), and max-file only takes effect when max-size is set - which is itself the evidence for the "no cap means it can fill the disk" premise. The outer shell passes sh -n, and I ran the classification against a stand-in that mimics docker's output structure: all three container states (normal, uncapped, capped-but-silent) classified correctly.

---

5. Is the application writing its output to a file, bypassing that cap?

grep -rn 'stdout_redirect' config/ | grep -vE ':[[:space:]]*#'

• Bad: a line redirecting stdout/stderr onto files in a log directory. This is the cause of "capped but 0 lines" above, and it is catchable at code review, long before the disk fills.
• Good: no results. Output goes to stdout, where the container's cap can actually govern it. Confirm separately that your production config points the logger at $stdout.

Actually run, and validated against the real before-and-after: clean on the current commit; check the same repo out at the commit before the fix and run it again, and it hits that exact stdout_redirect line. This is a check proven to fail.

Read the exit code the right way round here. This is grep, so a match - the fault being present - exits 0, and no match - the healthy case - exits 1. I measured both. If you wire this into CI under set -e without inverting it, you get a green build exactly when the fault is there, which is this article's whole thesis reproduced inside a four-word command.

In a Rails project the thing to look for is Puma's stdout_redirect. In any other framework, look for the same shape: any setting that attaches process output to a file.

---

6. Do database backups exist at all, and is the newest one recent?

ls -lt "$DIR"/*.sql.gz | head -5
n=$(find "$DIR" -maxdepth 1 -name '*.sql.gz' -mtime -2 2>/dev/null | wc -l)
echo "fresh backups: $n"; [ "$n" -gt 0 ]

• Bad - treat all three as failure: the directory does not exist; the directory exists but contains not one .sql.gz (the doc described a schedule that was never actually installed); or files exist but the newest is weeks old - the schedule is still there and the job has been failing quietly for a long time.
• Good: files are listed and the second number is greater than zero. Separately confirm the schedule is really installed: systemctl list-timers (Linux) or crontab -l.

Actually run, all four states verified with their exit codes: healthy (0), missing directory (1), empty directory (1), and a stale backup made by back-dating with touch (1).

That second line is written as a test rather than as a bare find | wc -l for a specific reason: wc is the last command in that pipe, so it exits 0 whatever it counts. Printed as a plain pipeline, the stale case - the one you most want to catch - prints 0 and reports success. Explicitly comparing the count is what makes the exit code mean anything.

DIR is deliberately a required argument rather than a hardcoded default - a production backup path does not belong in an article.

---

7. Does this dump actually contain data, or is it a very convincing empty file?

gzip -t "$D" \
 && gzip -dc "$D" | tail -5 | grep -c 'PostgreSQL database dump complete' \
 && gzip -dc "$D" | awk '/^COPY .* FROM stdin;$/{c=1;next} c&&/^\\\.$/{c=0;next} c{n++}
     END{print n+0; exit(n>0?0:1)}'

• Bad: the dangerous case is the first two passing and the row count being 0 - valid gzip, valid SQL, pg_dump's completion marker present, and not one row of data. File size will not save you: measured, the dump with data was 802 bytes and the empty one 785 bytes - a difference of 17 bytes. Check on size or existence and this backup sails straight into the retention queue.
• Good: gzip intact, marker count 1, and that final row count clearly greater than zero. All three, or it doesn't count.

Actually run, using real pg_dump output (local PostgreSQL 17) across three fixtures: a dump with rows (3 rows, exit 0), an identical-schema dump with no rows (0 rows, exit 1), and a truncated file (gzip reports corruption, exit 1). The awk state machine counts within each COPY block up to its \. terminator and depends on no external tool.

The exit(n>0?0:1) in that END block is load-bearing and was added after measuring. Without it awk prints 0 and exits 0, so the exact case this item exists to catch - the convincing empty dump - reports success. A check that prints the bad news and then returns "fine" is the failure mode this whole article is about.

---

8. Has anyone ever actually restored one?

trap 'dropdb "$S" 2>/dev/null' EXIT INT TERM
createdb "$S"                                              || exit 1
gzip -dc "$D" | psql -q -v ON_ERROR_STOP=1 -d "$S" >/dev/null || exit 1
n=$(psql -Atd "$S" -c "select coalesce(sum(cnt),0) from (
      select (xpath('/row/c/text()', query_to_xml(
        format('select count(*) as c from %I.%I', schemaname, tablename),
        false, true, '')))[1]::text::bigint cnt
      from pg_tables where schemaname='public') t")
echo "restored rows: $n"
[ "$n" -gt 0 ] || exit 1

• Bad: the restore succeeds and the total row count is 0. This is a different failure from the previous item: that one checks the file, this one checks what is left after the file goes back into a database. ON_ERROR_STOP=1 is the load-bearing part - without it psql swallows errors and exits 0, and you get a partial database plus a success report.
• Good: no errors during restore, and the printed total is greater than zero. This is the only step that turns "backup" from an assumption into a fact.

Actually run, both fixtures: the dump with data restored to 3 rows (exit 0), the empty one to 0 rows (exit 1). The trap ... EXIT INT TERM means the scratch database is dropped on every path out, including a Ctrl-C; I confirmed afterwards that pg_database had zero leftovers.

Same trap as the previous item, one level up. Written as a plain && chain ending in dropdb, the block's exit status is dropdb's - so a restore that produced zero rows, and even a restore whose row count you never looked at, exits 0. Capturing the count into a variable and testing it is the difference between a check and a printout.

Restore into a fresh throwaway database. Never run this against production. If your database lives in a container, the same shape is piping gzip -dc output into docker compose exec -T db psql.

---

9. Is there a destructive command you could hit by accident?

awk '/^[A-Za-z0-9_.-]+:/{split($0,a,":");t=a[1]}
     /--volumes|volume[ \t]+(rm|prune)|rm -rf|db:drop|db:reset/{
       if($0!~/^[ \t]*#/) printf "%s (line %d): %s\n", t, NR, $0 }' Makefile

• Bad: the destructive line sits under a harmless-sounding target (clean, reset, tidy). That is the actual risk: at midnight with a full disk, the operator scans make help for something that frees space and deletes production. The name has to match the consequence.
• Good: either no results, or every hit sits under a target whose name is shouting that it deletes things, behind a typed confirmation.

Actually run, and proven effective against the real before-and-after: against the pre-fix Makefile it printed clean (line 152); against the current one, the same line of code appears as destroy-all-data (line 383).

Same command, same line, and the only difference is whose name it sits under. This check does not answer "where are the dangerous commands" - it answers "what name is the dangerous command hiding behind", and that is what determines whether it gets triggered by accident. (The script accepts files under scripts/ too.)

---

10. Is this checkout on the commit you think it is?

git fetch -q
L=$(git rev-parse HEAD); R=$(git rev-parse @{u})
[ "$L" = "$R" ] && echo "ok: $L" || echo "BEHIND by $(git rev-list --count HEAD..@{u})"

• Bad: behind by N commits. This is the cheapest and most common silent no-op deploy: git pull failed on authentication and did not actually update, but every subsequent step ran anyway and reported success, redeploying the old code.
• Good: the two SHAs match. Also glance at git status --porcelain - uncommitted changes mean what is running corresponds to no commit at all.

Actually run, both directions: ok against this repo; and after cloning to a scratch directory and resetting back three commits, it correctly printed BEHIND by 3 and exited 1, with an added uncommitted change correctly producing an extra warning. All of that on the clone.

---

11. Was the artefact in production built from the commit you wanted?

intended=$(git rev-parse HEAD)
running=$(docker container inspect --format '{{.Config.Image}}' "$C")
echo "intended $intended"; echo "running  $running"
[ "${running##*:}" = "$intended" ] && echo MATCH || echo MISMATCH

• Bad - two ways:
  • MISMATCH: the running image was built from a different commit. Everything was green and the new code did not ship.
  • The tag is :latest: which means this is unanswerable, because a floating tag promises no commit at all. For this check to be possible, deploys have to be pinned to an immutable tag (the commit SHA).
• Good: the running tag is the intended commit SHA. This is the only check that separates "the deploy succeeded" from "the new code is live".

Not run locally (shape-checked against docs). docker container inspect --format is confirmed against the official docs, which also document Config.Image as an available field. .Config.Image is the image reference as requested (repo:tag), which is exactly what you want to compare against a commit (.Image gives the image ID and is not usable here). The outer shell passes sh -n, and I exercised match, stale and floating-:latest against three stand-ins producing docker's output structure; exit codes correct in all three.

---

12. Does the site actually work, or does it merely return 200?

The most important item on the list.

origin=$(printf '%s' "$PAGE" | sed -nE 's#^(https?://[^/]+).*#\1#p')
curl -sSL "$PAGE" | tr '<' '\n' | grep -i '^img ' \
 | sed -nE 's/.*src="([^"]+)".*/\1/p' \
 | sed -e 's#^//#https://#' -e "s#^/#$origin/#" | head -5 \
 | while read u; do
     b=$(mktemp)
     c=$(curl -sSL -o "$b" -w '%{http_code}' "$u")
     echo "$c $(wc -c <"$b") $(dd if="$b" bs=1 count=4 2>/dev/null | od -An -tx1) $u"
     rm -f "$b"
   done

• Bad: the page itself returns 200, but following one of its <img> to the end returns bytes starting 3c 68 74 6d (<htm) - that is an error page wearing a .png filename. This is exactly the class of fault in section 2: every page 200, every user-uploaded image broken. The page is fine; what is broken is what is inside it, and a status-code check can never see that.
• Good: every row is 200, with a plausible byte count and a real image magic number: 89504e47 is PNG, ffd8ff JPEG, 47494638 GIF, 52494646 WebP.

Actually run. This is also the only item where verification made me rewrite my own check. The first version tested Content-Type and my own fixture caught it out: a static server derives Content-Type from the extension, so a .png full of HTML still reports image/png and the check false-passes. You must read the actual bytes.

Verification: a local fixture site whose page returns 200 with one image that is an error page disguised as .png - the check caught it and exited 1; after fixing that image the same page passed and exited 0. Then run against real public sites to confirm it copes with real HTML.

The origin line is there because of a second thing the fixture caught. Most real pages write src="/path/img.png", and curl rejects that outright - "no host part in the URL" - so an earlier version printed a 000 status for every perfectly healthy image on the page. A check that fails everything is as useless as one that passes everything; resolving root-relative sources against the page's own origin is what makes it usable on a real site.

One caution: legitimate but tiny images (tracking pixels) should not count as failures. I downgraded that case to a warning, or this check will waste an evening of your life.

---

13. Are the share-preview tags absolute URLs that actually resolve?

v=$(curl -sSL "$PAGE" | tr '<' '\n' \
    | grep -iE '^meta[^>]*property="og:image"' \
    | sed -nE 's/.*content="([^"]*)".*/\1/p' | head -1)
case "$v" in
  https://*|http://*) echo "absolute: $v" ;;
  *)                  echo "RELATIVE (broken): $v" ;;
esac
curl -sSL -o /tmp/og -w 'http %{http_code} bytes %{size_download}\n' "$v"
dd if=/tmp/og bs=1 count=4 2>/dev/null | od -An -tx1

• Bad: a relative path. It looks completely fine in a browser and raises no error anywhere, but Open Graph requires an absolute URL, so every social platform renders no preview. The other case is absolute but 404 - the fallback image never existed. Neither will report anything to you.
• Good: an absolute https:// URL, 200, and a real image magic number. Check twitter:image and og:url the same way.

Actually run, both directions: a fixture using a relative path was correctly flagged RELATIVE and exited 1; switched to an absolute URL with the image present, it passed. Then against a real public site it correctly extracted the absolute URL, 200, 36,471 bytes, magic 89504e47 (PNG).

Check the fallback path separately from the per-page path - they fail independently, and the fallback covers most of your pages. And it is worth knowing why "relative path plus summary_large_image" is especially bad: that layout is almost entirely image, so empty looks worse than no card at all.

---

14. Is the stylesheet much bigger than it should be, and what is inside it?

f=app/assets/builds/application.css
echo "total $(wc -c <$f) | gzip $(gzip -c $f | wc -c) | inline sourcemap $(tr -d '\n' <$f | grep -o 'sourceMappingURL=data:[^*]*' | wc -c)"

• Bad: an inline sourcemap taking up a large fraction of the file. That is not CSS, it is a base64 map embedded by a build tool, and every visitor downloads it before first paint. Measure before you cut: reaching for tree-shaking without knowing what the size actually is, is far riskier - broken styles raise no error, they just quietly look wrong.
• Good: inline sourcemap 0, and a gzipped size that matches your intuition about how much styling this site has - the gzipped number is what visitors actually download.

Actually run, both directions. Current build output: 472,411 bytes, gzip 61,844, inline sourcemap 0 - and those two figures match the post-fix numbers in the commit message exactly, so they are traceable. The bad case was reconstructed from the current CSS plus an embedded base64 sourcemap; the detector reported 1,389,802 bytes with 66% non-CSS, very close to the real pre-fix 1,393,531 bytes / 65.9%.

The gzipped number from the reconstruction is not quotable - random base64 does not compress like a real sourcemap does. The real before/after gzip figures are 183,803 → 61,844.

---

14. What is still not fixed, and what was deliberately deferred

If this article only listed what got fixed, it would be marketing. So:

Deliberately deferred:

• Bootstrap tree-shaking. The CSS is now 472,411 bytes (61,844 gzipped) and is still larger than it needs to be. The stated reason for not doing it: broken styles raise no error, they just quietly look wrong, and on a low-traffic site that can go unnoticed for weeks. Take the zero-risk --no-map win that removed the 66%; leave the rest until someone actually measures visual regression.
• docker image prune -a stayed out of clean, because -a deletes the current app image whenever the stack happens to be down, forcing a full rebuild on a host that cannot afford one. A deliberately less thorough cleanup.

Known and not yet done:

• Four files in this repo still assert the wrong "all 1,778 rows were identical" conclusion (section 8). The correct figure lives in the rake task and the correcting commit. I am writing about it rather than quietly fixing it first, because it is the article's own thesis: a finding written down before it was verified propagates to four places, and no automated check will ever tell you a comment is wrong.
• Deploy-time checks are not alerting, and this audit did not turn them into alerting. A preflight threshold that refuses an operation which cannot finish, and a df printed after a deploy, are real improvements - but they only speak when you deploy. Continuous alerting on the slow variables is a separate piece of work with its own design, and section 6 is the argument for doing it: that fault class degrades silently for months before it does anything you can see.
• A verified backup and a durable backup are two different problems. Going from "none at all" to "scheduled, verified, restorable" is the step this audit took, and it is the step that turns an assumption into a fact. How many copies exist and how far apart they sit is the next question, and a working nightly dump answers none of it. Do not let the first one convince you that you have solved the second.

Two honest notes about the audit itself:

• The 127 Errno::ENOSPC occurrences are the weakest-sourced figure on the list. They appear in three places in the repo, but all three are the same author recalling it, and there is no log artefact. Month precision, and that is all.
• "89,135 backtraces filled the disk" is an inference, not a measurement. Two independent sources say respectively that those backtraces were the main cause of the disk filling, and that the log volume reached 965MB. They are almost certainly the same bytes - but no single source says so.

---

In one sentence

Not one of these ten faults was caught by checking whether the site was up, because the site was up the whole time.

They all lived in the same gap: between "something is running" and "the right thing is running." And almost every off-the-shelf check - status codes, health endpoints, exit codes, list-timers, docker inspect - measures the left half of that gap.

So the most useful rule I took away is this: for every check you depend on, ask what question it actually answers, then ask what question you meant to ask. The place where those two differ is where your faults live.

And the one from the repo that I keep using:

A check never proven to fail is not a check.

Pick one item from the list above tonight, then break it on purpose and confirm that it screams.

---

Every figure in this article comes from this production server's git history, a code comment, or output I measured myself between 4 and 6 August 2026. Two of them I independently re-measured while writing, and they matched the commit messages exactly (the CSS at 472,411 / 61,844 bytes, and the 1200x630 PNG). Where something is an inference I have labelled it as one. Four Docker-related checks were not executed locally and are labelled "shape-checked against docs", because that is more honest than presenting a command nobody ran as tested.

For security reasons this article deliberately omits: rate-limit thresholds and windows, spam-detection signatures, the honeypot field name, the kill-switch variable name, credential file paths, the registry namespace, the deploy account, the checkout path, and the server's IP. For every defence I have written the shape and the reasoning, never the parameters.