I run a lot of systems, so things break. These write-ups follow a blameless format: what happened, who it hurt, how I fixed it, and what I changed so it doesn't happen again. Each "what changed" is the part I care about most.
53,000 API calls in one hour took the portal down
West Muskingum Local Schools, August 2026
What happened. A stuck n8n workflow kept calling WM.Tools' GAM endpoint in a loop: about 53,000 calls in one hour. Each call ran the Google Workspace CLI in a sibling container with a 60-second timeout. 1,360 of them hit that timeout.
Impact. All 50 PHP-FPM workers were blocked waiting on GAM while the CPU sat under 1%. The whole portal stopped answering for every staff member in the district, not just the automation.
Fix. I stopped the workflow, then changed how GAM is called:
- A site-wide cap of 5 concurrent GAM calls, built from cache-lock "slots" with a TTL so a crashed worker can't hold one forever
- Random 50–150 ms jitter between attempts
- Fail fast with "GAM is busy" after 10 seconds instead of queuing
- The default interactive timeout dropped from 60s to 25s
What changed. A flood can no longer eat the worker pool, because callers get rejected instead of piling up. The scheduled sync jobs were also staggered and locked so they never compete for the same slots.
The full design is in the WM.Tools case study.
A personal site was compromised through an outdated dependency
Personal site, September 2026
One of my sites was running an outdated Livewire version with a known vulnerability, and it was exploited. I rotated its credentials, then wrote an audit script that patched all 19 Laravel apps I host, turned off debug mode where it was on, and rolls an update back automatically if a site stops responding. Patching is now routine instead of "when I remember".
A power outage corrupted our team password vault
West Muskingum Local Schools, 2026
What happened. A power outage hit the server running Vaultwarden, our IT team's shared password vault. Its SQLite database came back corrupted.
Impact. Our shared credentials were unavailable until the vault was restored.
Fix. We restored from the previous day's backup. I had set up two independent backup paths:
- A weekly update script stops the container cleanly so SQLite checkpoints its write-ahead log, archives the data, and rolls back automatically if the new version fails its health check
- An n8n workflow takes nightly backups with a monthly long-term copy, and has a self-service restore form that promotes a chosen backup to live
What changed. Nothing, which was the point. The backups existed because I'd planned for exactly this, and the restore had a documented path. It's also my answer when someone asks "have you ever actually restored from a backup?"
Log4Shell on the Minecraft servers
Homelab, December 2021
What happened. The Log4j vulnerability (CVE-2021-44228) affected Java game servers, including the modded Minecraft servers I ran for a community of players.
Impact. A chat message could trigger remote code execution on the server.
Fix. I applied the published mitigations: the JNDI lookup fix and a patched Log4j configuration, on every server I ran.
What changed. It was my first real CVE response on a service other people depended on. It taught me to treat "it's just a game server" as a production service with real users.
The proxy VM keeps running out of disk
Homelab, ongoing
What happened. My nginx reverse-proxy VM has a 16 GB disk. Logs and temp files fill it up every so often.
Impact. When it's full, nginx can't write, and sites behind it start failing.
Fix (so far). I go in and clean out temp files and old logs by hand. That works, but it's a chore, not a fix.
What changes next. Proper log rotation and a scheduled cleanup, plus a disk-usage alert sent to my self-hosted ntfy before it gets full. I'm including this one because "I fix it by hand" is honest, and the plan to stop doing that is the point.