In short and in plain language
When the server runs out of space, you often see no clear outage. The shop keeps loading. But orders get stuck.
The article shows which data grows in a shop. It also shows how to monitor storage.
You set a limit. When it is reached, there is an alert. You also clean up at fixed intervals.
We monitor the storage of your shop and set a plan for cleaning up.
A full data partition rarely produces a clean outage. The shop keeps answering, the catalogue loads, the availability check reports HTTP 200 - and yet the database writes no further row, the order stays half finished and the confirmation never leaves the building. Disk space is therefore not a clean-up topic for the quiet week in December, it is a metric of its own with a threshold, an alert and a fixed cycle. This article covers the four sources that grow unnoticed in a shop, why the threshold belongs in percent rather than in gigabytes, how a full disk differs from a reached quota and which cleanup routines belong in the maintenance plan.
Key takeaways
- When the data partition fills up, the shop reports no outage: it keeps answering and loses writes. The database does not abort in that state, it checks once every minute (MySQL 8.4 Reference Manual) whether there is space again.
- The four silent growth sources are binary logs, session data, cache and temporary files, and generated documents including image derivatives. Their defaults are fixed, but nobody set them for this shop: binary logs are kept for 30 days (MySQL 8.4 Reference Manual) out of the box.
- The threshold belongs in percent, because every disk upgrade silently shifts a value given in gigabytes. On top of that comes a reserve the shop never sees: ext4 sets aside 5% (mke2fs(8)) of capacity for the super-user when the file system is created.
- A full disk and a reached quota look identical in the error pattern and call for different answers. Only that distinction turns an alert into an action, and only a dated cleanup cycle turns the action into a routine.
Why a full disk is not an outage
An outage has one advantage: it is noticed. The page stops answering, the availability check fires, somebody gets up. A full data partition behaves differently, because reading needs no space: the catalogue appears, search answers, the basket fills. Only writing - creating an order, updating stock, recording a payment state, saving a session - changes the behaviour, and not with an error page but with waiting.
The database does exactly what its documentation describes. If the disk fills up during a write, the server waits instead of aborting and checks once every minute (MySQL 8.4 Reference Manual) whether space has become available. In parallel it writes a note to its error log every 10 minutes (MySQL 8.4 Reference Manual). Tables are not the only thing affected: the documentation explicitly names writes to the binary log files (MySQL 8.4 Reference Manual) and to the binary log index file. Anyone sitting in the checkout during that minute sees a loading bar.
The alert that never arrives
The four sources that grow unnoticed
Storage rarely grows where it is expected. Product images and file attachments are visible, get planned and are noticed immediately if they get out of hand. The four sources that actually fill a partition are invisible, because they belong to normal operation: the database binary logs, session data, cache and temporary files, and generated documents such as invoices and image derivatives. None of them shows up in an inventory that looks for large files, because all four consist of many small ones.
The binary logs are the quietest and at the same time the hungriest item. They record every change to the data and are the basis for replication and for point-in-time recovery - nobody wants to switch them off. By default they are kept for 30 days (MySQL 8.4 Reference Manual), and they are removed at startup and when the binary log is flushed (MySQL 8.4 Reference Manual), not continuously. A shop with lively stock synchronisation therefore writes a multiple of its actual data volume onto the same partition over a month.
| Source | Default according to the documentation | What makes it grow | What belongs in the maintenance plan |
|---|---|---|---|
| Database binary logs | Retention 30 days (MySQL 8.4) | Every write, independent of the size of the data set | Tie retention to the recovery objective, not to the default |
| Session data | Lifetime 1440 seconds, cleanup at 1 in 100 (PHP Manual) | Every visit, even without an order; crawlers count too | Own location, own cleanup run, own measurement |
| Cache and temporary files | No system-wide default, differs per application | Every catalogue rebuild and every regeneration; old states stay | Clean up after each build, use age instead of size as the criterion |
| Generated documents and image derivatives | No default, retention follows tax law | Linear with orders and articles, derivatives per format and width | Separate location, retention written down |
| System journal (journald) | 10% of the file system, capped at four gigabytes (journald.conf(5)) | With errors and with load; fastest during an incident | Set a fixed upper limit instead of letting the default decide |
The table shows the common denominator: none of these sources knows the shop. They only know their own default, and that default was written for some server, not for this server with this partition and this order volume. How closely database maintenance and storage demand are connected is covered in the article on database maintenance for shops.
The threshold belongs in percent, not in gigabytes
A threshold in gigabytes is a snapshot. It is correct for the partition it was set on and turns silently wrong with the next upgrade: "alert at 20 GB free" becomes, on a large disk, a warning that arrives an hour before the standstill, and on a small one a warning that is permanently active and therefore eventually switched off. A share in percent stays comparable across every upgrade and can be carried from one system to the next.
There is one detail that shifts the arithmetic. When the file system is created, ext4 sets aside 5% (mke2fs(8)) of capacity for the super-user; the shop never gets to use that share. On a pure data partition it is more reserve than necessity and can be lowered afterwards. The ratio of inodes to capacity is a different matter: it is fixed at creation time and cannot be changed afterwards (mke2fs(8)). A partition full of small files can therefore refuse another file while usage still shows 60 percent.
Two thresholds, two reactions
What the threshold triggers matters just as much. An alert that only says "disk almost full" creates a search. An alert that also carries the three largest directories, their growth since yesterday and the time of the last cleanup run creates a decision. The difference costs half an hour of setup once and saves the first quarter of an hour in every incident - exactly the one that emergency support otherwise spends building context.
Full disk or reached quota
In the error pattern both states look the same: writing fails or hangs. The cause differs, and so does the answer. A full partition affects everything stored on it - database, sessions, logs, documents. A reached quota affects exactly one account or one directory, while there is free space next to it and every measurement of the partition returns a healthy number.
On shared hosting the quota is the normal case and treacherous for precisely that reason: the file system has space, the account does not. Measuring only the partition in that situation shows a healthy number and a sick shop. Both metrics therefore belong in monitoring, each with its own threshold and its own recipient. The same separation applies to database quotas, mailbox quotas and the location of the backups - three places where space can run out independently of one another.
Binary logs
Retention follows the recovery objective, not the default. Anything kept longer than the backup strategy requires is pure storage consumption.
Session data
Lifetime and cleanup run are two separate settings. The first says when a session is old, the second whether anybody ever looks.
Cache and derivatives
Image derivatives are regenerated per format and width, and old states do not disappear on their own. How to keep that in check is covered in the article on WebP and AVIF in the shop.
Generated documents
Invoices grow linearly with revenue and are subject to retention rules. They belong in a separate location so that no cleanup run can ever reach them.
Temporary files
Aborted imports, half-written exports, leftovers from updates: the temporary directory is not a self-cleaning oven, it is a filing cabinet with a memory.
Route to a person
Every source needs a recipient. A value without an addressee is a number on a dashboard that nobody opens during an incident.
Six items, six responsibilities - that sounds like effort and is, in practice, a one-off setup. After that storage is a metric like response time or error rate, and it behaves more kindly than both: it changes slowly and announces itself. Measuring it buys weeks of lead time where there would otherwise be minutes.
Defaults that nobody ever set
Three examples show how defaults play out in operation. Session data counts as stale after 1440 seconds (PHP Manual), but it is only collected with a probability of 1 in 100 (PHP Manual) per request - with little traffic much is left behind, with heavy traffic the pile grows faster than chance removes it. The system journal takes 10% (journald.conf(5)) of the file system by default, capped at four gigabytes. Once that limit is reached it does not hand back space it already uses (journald.conf(5)), it merely stops taking more.
The thread is aborted the next time it checks the disk (in one minute).
That sentence describes the behaviour that reaches the shop as "slow". The connection stays open, the job is neither rejected nor completed, and when the process is finally aborted, half a transaction is left in the database. Exactly those half states reappear later as a missing payment, a duplicated order or as silent background job failures - by then without any visible connection to the disk.
#!/bin/bash
# Report capacity per partition and per quota. Thresholds in percent, not in gigabytes.
# Called from the maintenance cycle, output to a log file, exit code drives the alert.
TARGET="/var/www/shop"
WARN=80
CRITICAL=90
RC=0
# Used share of the partition, as the shop sees it (without the super-user reserve).
used=$(df --output=pcent "$TARGET" | tail -n1 | tr -dc '0-9')
echo "partition_used_percent=$used"
if [ "$used" -ge "$CRITICAL" ]; then
echo "ALERT partition $TARGET at $used percent - on-call"
RC=2
elif [ "$used" -ge "$WARN" ]; then
echo "WARNING partition $TARGET at $used percent - next business day"
RC=1
fi
# Name the three largest directories so the alert already carries an action.
du -x -m -d 1 "$TARGET" 2>/dev/null | sort -rn | head -4 | while read -r mb path; do
echo "directory: $mb MB $path"
done
# Measure the quota separately: the file system may have space, the account may not.
if command -v quota >/dev/null 2>&1; then
quota -w -s 2>/dev/null | tail -n +3 | awk '{print "quota:", $1, "of", $2}'
else
echo "quota: no quota tool present - value not measured"
fi
# Oldest binary log: if it reaches beyond the configured retention, the setting is not taking hold.
oldest=$(find /var/log/mysql -name 'binlog.*' -printf '%T@ %p\n' 2>/dev/null | sort -n | head -1)
echo "binlog_oldest=${oldest:-not measured}"
exit "$RC"
The fixed cleanup cycle
Cleaning up on demand is not a procedure, it is the reaction to an alert that should never have happened. A cycle has three properties: it is dated, it is logged, and for each source it carries a criterion that does not read "too big" but names an age or a count. Only then does it stay correct when traffic doubles.
- Weekly: record usage per partition and per quota, note the growth against the previous week. A single number says little, the slope says everything.
- Weekly: check binary log retention against the recovery objective. Anything kept longer than recovery reaches back is ballast - see disaster recovery for shops.
- Monthly: clear the session store and the temporary directory by age, not by size. The criterion then stays correct as traffic grows.
- Monthly: reconcile image derivatives against the article inventory; derivatives of deleted articles otherwise stay forever and nobody misses them.
- Quarterly: reconcile retention obligations with the actual storage layout so that no cleanup run can reach documents that must be kept - details in the article on GDPR-compliant backup retention.
- After every build and every larger import: check cache states and leftovers before the maintenance window closes and nobody looks any more.
The cycle belongs on the same list as updates and backups, which means it belongs in the maintenance contract. Not for formal reasons, but because tasks without a fixed slot in the calendar are dropped in exactly the week in which they would have mattered.
From alert to relief
Measure before deleting
Usage, growth since the previous week, the three largest directories and the state of the quota. Deleting first loses the trail and repeats the same incident four weeks later.
Attribute the growth
An import that leaves remnants behind every night looks like a full log but is a different fault. The attribution decides whether the answer is "clean up" or "switch off".
Free space safely
Start with what is clearly dispensable: expired sessions, old cache states, leftovers in the temporary directory. Binary logs are removed through their retention setting, never by hand from the directory, otherwise the chain for point-in-time recovery breaks.
Record the cause
Log the threshold, the time, the trigger and the measure in one sentence. After three incidents the log shows whether the partition is too small or the cycle too rare - two questions with very different answers.
Usage, growth since the previous week, the three largest directories and the state of the quota. Deleting first loses the trail and repeats the same incident four weeks later.
An import that leaves remnants behind every night looks like a full log but is a different fault. The attribution decides whether the answer is "clean up" or "switch off".
Start with what is clearly dispensable: expired sessions, old cache states, leftovers in the temporary directory. Binary logs are removed through their retention setting, never by hand from the directory, otherwise the chain for point-in-time recovery breaks.
Log the threshold, the time, the trigger and the measure in one sentence. After three incidents the log shows whether the partition is too small or the cycle too rare - two questions with very different answers.
The order is not arbitrary. The most expensive mistake in this situation is quickly deleting binary logs in the file system: space is available afterwards, point-in-time recovery is not. A backup whose restore is only tested in an emergency is not a backup but an assumption; how to demonstrate the difference is covered by the backup service.
What retention demands
Not everything that costs space may be deleted. For records relevant to taxation, Section 147 subsection 3 of the German Fiscal Code stages the periods: books and records must be kept for ten years (German Fiscal Code), accounting vouchers for eight years and the other documents listed there for six years. For a shop that means invoices and accounting vouchers are not a candidate for a cleanup run but an item whose growth is planned for and whose location is chosen so that no routine reaches it.
The opposite direction matters just as much: whatever is not subject to retention needs an upper limit. MariaDB ships a different default for binary log retention than MySQL. There the default value is 0 (MariaDB Server Documentation), meaning no time limit at all, the permitted range reaches up to 8553600 seconds (MariaDB Server Documentation), and removal happens at startup and at binary log rotation (MariaDB Server Documentation). Copying the values of one system from a guide written for the other yields either a partition full of logs or a gap in recovery.
A metric, not a feeling
Usage and growth per partition, collected in the same cycle as response time and error rate. Only the slope turns a number into a forecast.
Separate locations
Database, logs, documents and cache on one partition means one outlier stops everything. Separated means one outlier stops one thing.
A dated cycle
Every cleanup routine has a date, a criterion and an entry in the log. Without the entry there is no way to tell later whether it ran or was skipped.
Retention protected
Documents that must be kept live where no cleanup run reaches. That is a question of storage layout, not of care in the individual case.
Alerts with context
The message names the three largest directories and the growth since yesterday, otherwise every incident starts with the same search.
Capacity planned ahead
The growth of the last three months gives the date of the next expansion. It can only be planned while it is not yet urgent.
Together the six points are not a project but a routine. In monitoring, storage has the best ratio of effort to effect: cheap to collect, slow to change and able to prevent an incident that would otherwise reach the checkout before it shows up on any dashboard.
Capacity in the maintenance cycle
Disk space is the least conspicuous metric in a shop and one of the few that announce an incident weeks in advance. Measuring it in percent, setting an upper limit per source, dating the cleanup cycle and looking at the quota separately turns a standstill at four in the morning into a task on Tuesday morning. The rest is bookkeeping: which partition grows how fast, which retention is mandatory, when the next expansion is due. How closely capacity is tied to other silent failure patterns is shown by the article on stale cache in the shop and by the analysis in checkout monitoring.
Sources
Related Articles
Monitoring indexability: when noindex reaches production
A noindex carried over from staging is often noticed only when revenue drops. Here is a five-stage check chain that reports deviations on the same day.
WordPress security updates: what old branches still get
WordPress 7.1.2 closed a critical flaw and the fix was backported to 4.7 as a courtesy. What older branches receive, what they do not, and when to move up.
Discount codes in Black Week: detecting abuse early
Discount codes in Black Week: code length, a rate limit at the coupon field, alerts on failed attempts and new-customer discounts that resist multiple accounts.