A report request returns successfully. The job enters a queue. A worker fails before writing the file. From the browser’s first response, everything looks healthy; from the user’s perspective, nothing arrived. Monitoring needs to be able to tell those two stories apart.
Track the steps that matter
Use a fictional report-export service to map the distinction. The application accepts a valid request, the queue holds it, a worker processes it, storage receives the file and the user can retrieve the result. Each checkpoint has a different owner and a different kind of failure.
Give the operation a durable identifier that can connect those checkpoints. Log state changes and relevant error categories without copying the report’s contents into ordinary telemetry. The identifier should help an operator find a failed operation, not become an alternative route for an unauthorized person to download it.
Choose useful monitoring signals
A queue-length graph can reveal pressure, but the oldest unfinished job may better answer whether users are waiting too long. A successful worker invocation does not establish that the file exists. A stored file does not establish that the intended user can retrieve it. Pick the measurement that corresponds to the decision you need to make.
Google’s SRE monitoring chapter separates internal measurements from externally visible behavior and organizes service observation around latency, traffic, errors and saturation. It also emphasizes alerts that can be acted on. Apply that distinction to the export journey rather than reproducing every available graph.
Test common failure scenarios
Test with synthetic reports. Pause a worker, reject a storage write and remove the requesting user’s download permission. Those exercises should not all produce the same diagnosis. The operator needs to know whether to restore processing, fix a destination or investigate an authorization decision.
- Accepted but waiting: identify the operation and the component holding it.
- Processing failed: distinguish a retryable failure from invalid work.
- Completed but inaccessible: check the stored result and the current access decision separately.
Give each alert a clear next step
Write down what the recipient should inspect first and which action they are allowed to take. Blindly retrying an export may create duplicate files; treating every access denial as an outage may encourage someone to weaken a correct control.
A useful dashboard lets the on-call engineer explain where the user journey stopped. Start with that explanation. Add more telemetry when it resolves a question the existing signals cannot answer.