An online shop produces traces around the clock: every order, every payment, every failed login and every slow database query leaves a log entry. But unused logs are just data clutter on a disk. Only structured logging, central aggregation and well-designed alerting turn those traces into early warnings. The Google SRE Book describes the four Golden Signals (latency, traffic, errors, saturation) as a proven foundation for exactly this (Google SRE Book). This guide shows how to structure logs meaningfully, set alerts without driving your team into alert fatigue, and respond quickly when it counts. The matching technical foundation is delivered by professional log monitoring.
Structured Logging as the Foundation
Classic log lines are written for humans: a timestamp, a message, maybe a stack trace. They are unsuitable for machine analysis because every component uses its own format. Structured logging solves this by writing each entry as a machine-readable object -- usually JSON -- with clearly named fields: timestamp, log level, message, service, request ID and business context such as order_id or customer_id. Logs can then be filtered, aggregated and correlated rather than merely searched.
The OWASP Logging Cheat Sheet recommends recording security-relevant events consistently and tamper-resistantly -- for instance failed logins, permission changes or input validation errors (OWASP Logging Cheat Sheet). Equally important is what is not logged: passwords, full credit card numbers or session tokens must never end up in a log. For German shops this is also a data protection obligation; the BSI assigns appropriate logging to module OPS.1.1.5 of its IT-Grundschutz framework (BSI IT-Grundschutz). Where possible, personal data is pseudonymized in the log.
{
"timestamp": "2026-06-03T14:22:08.512Z",
"level": "ERROR",
"service": "shopware-checkout",
"trace_id": "a91f3c7e-1d22-4f0b-9c11-8e2a",
"route": "/checkout/finish",
"http_status": 500,
"order_id": 10493,
"customer_ref": "cust_7f12a",
"message": "Payment provider timeout after 8000ms",
"error_code": "PAY_TIMEOUT"
}One request ID that connects everything
trace_id per incoming request and pass it through every layer -- from the web server through the Shopware application to the payment gateway. That lets you reconstruct a single failed checkout across all services instead of digging through five separate log files.Using Log Levels Correctly
Log levels are the basis of any meaningful filtering. Logging everything at the same level drowns important messages in noise. A proven hierarchy helps keep an overview day to day and quickly find the relevant entries during an incident. Consistency is key: an ERROR must mean the same thing in every service, otherwise thresholds lose their meaning.
DEBUG / TRACE
Detailed information for development. Usually disabled in production or enabled only temporarily to narrow down a specific problem.
INFO
Business events in normal operation: order created, payment confirmed, cron job finished. The basis for business metrics derived from logs.
WARN
Anomalies that are not yet outages: slow responses, repeated payment attempts, approaching memory limits.
ERROR
A specific operation failed: checkout aborted, API call unanswered. A prime candidate for threshold-based alerts.
CRITICAL / FATAL
The service or a core function is no longer usable. Such events justify immediate escalation to the on-call service.
SECURITY / AUDIT
Keep security and audit events separate: logins, permission changes, admin actions -- traceable and tamper-resistant.
Central Log Aggregation
Logs on individual servers are worthless when it matters: anyone who has to SSH into each server during an incident loses precious minutes -- and on a crashed server may not reach the logs at all. Central log aggregation collects all entries via a collector agent, normalizes them into a unified schema and makes them searchable. Open-source tools handle collecting, parsing and indexing; the concrete choice depends on data volume and hosting environment.
Three aspects are decisive here. First, retention: how long are logs kept? A few weeks are often enough for troubleshooting, while security-relevant audit logs should be retained longer. Second, volume: unthrottled DEBUG logging can quickly generate several gigabytes per shop per day, which is why production filters deliberately. Third, access protection: logs often contain sensitive information and must be secured accordingly. The central setup addresses all three points from the start.
# Example collection pipeline (generic, open source)
sources:
nginx_access: { path: /var/log/nginx/access.log, parser: json }
shopware_app: { path: /var/www/shop/var/log/*.log, parser: json }
mysql_slow: { path: /var/log/mysql/slow.log, parser: slowlog }
transforms:
enrich:
add_fields: { env: production, shop: shop-01 }
drop_fields: [ password, card_number, session_token ]
sinks:
central_store:
index_by: [ service, level, http_status ]
retention_days: { default: 30, security: 365 }Metrics, Logs and Traces Working Together
Logs are only one of three pillars of observability. The CNCF project OpenTelemetry describes metrics, logs and traces as complementary signals that only together form a complete picture (CNCF OpenTelemetry). Each pillar answers a different question, and the biggest leverage comes when they are linked via a shared trace_id: a metric shows that something is wrong, the associated log explains why, and the trace shows where in the call chain.
| Signal | Answers | Typical shop example | Strength for alerts |
|---|---|---|---|
| Metrics | What is happening, in numbers? | Error rate, p95 latency, orders per minute | Very good: cheap to store, ideal for thresholds |
| Logs | Why did it happen? | Stack trace of an aborted checkout | Good: precise root cause analysis, higher volume |
| Traces | Where in the system does it stall? | Request across app, database and payment gateway | Strong for distributed calls and latency issues |
For most online shops a pragmatic entry point makes sense: first clean metrics for the most important figures, then structured logs for root cause analysis, and finally tracing for the bottlenecks that logs alone cannot explain. It is important to base alerts primarily on metrics and use logs for the subsequent diagnosis -- this keeps alerts stable and log volume manageable.
Meaningful Alerts: Thresholds, Error Rates, Anomalies
An alert is only worth its noise if it points to something a human actually needs to fix. The Google SRE Book offers a clear rule for this: alerts should be symptom-based and target the impact on the user, not every individual technical cause (Google SRE Book). For a shop this means an alarm on a rising checkout error rate is more valuable than a hundred individual messages about brief CPU spikes.
- Thresholds: Fixed limits with a time window, such as a 5xx error rate above 2 percent over five minutes. The time window prevents alarms from single outliers.
- Error rates and error budgets: Look at the relative error ratio instead of absolute numbers. The DORA program shows that high-performing teams steer reliability through such measurable goals (DORA State of DevOps).
- Latency percentiles: It is not the average that counts but p95 or p99. A good average can hide that every twentieth customer experiences a painfully slow checkout.
- Anomaly detection: Report deviations from the expected pattern -- such as zero orders during a window where orders normally arrive. Such business alerts catch outages that look technically inconspicuous.
- Saturation: Flag approaching limits such as memory, connection pool or disk early, before they turn into an outage.
groups:
- name: shop-checkout
rules:
- alert: HighCheckoutErrorRate
expr: sum(rate(http_requests_total{route="/checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{route="/checkout"}[5m])) > 0.02
for: 5m
labels: { severity: critical }
annotations:
summary: "Checkout error rate above 2% for 5 minutes"
runbook: "https://wiki.internal/runbooks/checkout-5xx"
- alert: NoOrders
expr: sum(increase(orders_completed_total[15m])) == 0
for: 15m
labels: { severity: critical }
annotations:
summary: "No completed order for 15 minutes"Every alert needs a runbook
Avoiding Alert Fatigue
The biggest risk of an alerting system is not the missing alarm but the too frequent one. Anyone constantly receiving notifications that require no action starts ignoring them -- and eventually overlooks the decisive alarm too. In practice an estimated 73 percent of teams report some form of alert fatigue (project experience). The Google SRE Book is explicit here: every alert that wakes a person must be urgent, actionable and not automatically resolvable (Google SRE Book).
- Alert only on symptoms that affect the user -- not on every individual technical metric.
- Separate severity levels clearly: what wakes someone at night, what waits until morning?
- Deduplicate duplicate alarms and group related messages into a single incident.
- Mute known maintenance windows to suppress predictable alarms.
- Review alert rules regularly and remove rules that never led to an action.
- Fix recurring, automatically resolvable problems at the root instead of alerting permanently.
Escalation and On-Call
A precise alarm is only useful if it reaches the right person and keeps going if in doubt. An on-call rotation defines who is responsible for alarms at which time, and an escalation chain ensures that an unacknowledged alarm automatically reaches the next level. A proven model for a shop: Level 1 immediately as a chat and push message, Level 2 after five minutes as an SMS, Level 3 after fifteen minutes as a call to the on-call service, Level 4 after thirty minutes to the technical lead.
For on-call to work sustainably, the load must remain bearable. The Google SRE Book recommends limiting on-call load and nighttime interruptions so that being on call does not become a permanent burden (Google SRE Book). A small, well-rehearsed rotation with clear handovers and documented responsibilities is more reliable than a large distribution list in which nobody ultimately feels responsible. For many shop operators it therefore makes sense to hand nighttime readiness to an external service provider.
Escalation chains must be tested
Incident Response: From Alarm to Resolution
When an alarm becomes a real incident, the quality of the response determines the extent and duration of the damage. A central metric here is the Mean Time to Recovery (MTTR) -- the average time from detection to restoration. The DORA program counts recovery time among the core metrics of high-performing software teams (DORA State of DevOps). The goal is not to avoid every error but to fix errors quickly and in a controlled way.
- Detect and classify: Acknowledge the alarm, check severity, determine the affected function (is checkout affected or only a peripheral area?).
- Diagnose: Use the
trace_idto combine related logs, metrics and traces and narrow down the cause. - Stabilize: Limit the impact first -- for example via a rollback or disabling a faulty feature -- before fixing the final cause.
- Communicate: Inform those affected internally and, if necessary, customers transparently, ideally via a status page.
- Review: A blameless postmortem records cause, course and improvements -- turning every incident into a lasting learning gain.
Hope is not a strategy. Reliable systems come from measurement, clear signals and practiced response -- not from the wish that nothing will happen.
Once logging, aggregation, alerting and incident response are set up cleanly, the same structure can be used across all updates and releases. Closely intertwined with this is solid release discipline: how to roll out changes with low risk and catch errors beforehand is described in the article on staging environments and safe updates. Together, these two building blocks -- good operational visibility and controlled changes -- form the backbone of a reliably running online shop. Our standby and emergency support runs this chain for you day to day; for acute incidents, the direct line to the team supports you.