Methodology · Measurement · Methodology
How RELIASTRA measures vendor reliability
The measurement methodology behind every check: regional origination, retry semantics, quorum rules, and the cases we deliberately refuse to call an outage.
Every number RELIASTRA shows comes from an HTTP request that something made, at a recorded time, from a recorded place. This page describes how those requests are scheduled, how the results are interpreted, and - more importantly - the cases we deliberately refuse to call an outage.
1. Checks are scheduled, not sampled on demand
A single authoritative scheduler dispatches checks on a fixed interval per dependency. Each due dependency produces one independent task per configured region, published through a message broker and executed by a worker. Checks never run inside the API process that serves your dashboard, so a busy dashboard cannot delay a probe and a slow probe cannot block the API.
If the scheduler, the broker or a worker is unavailable, checks simply do not run - and the system reports that. A missed probe is never backfilled with a synthesised result, because an observation that did not happen must never appear in a history you intend to rely on.
2. Origins are separate and recorded
Each region resolves and connects independently, and every stored result carries the region that produced it. Two regions disagreeing is information: it distinguishes a vendor-wide failure from a network path problem affecting one origin.
3. Quorum, not a single failed request
A single failed request is not an incident. Declaring one requires a quorum - failures observed across more than one region inside a short correlation window - and recovery likewise requires consecutive successful checks before a dependency is marked healthy again. The window and the region count are fixed, small integers rather than tunable dials, so the same evidence produces the same verdict everywhere.
4. Targets are validated before they are probed
Before any request leaves, the target address is resolved and checked against a security policy that rejects private, loopback, link-local and metadata addresses. A target that fails that check is recorded as blocked by security policy, and no request is sent.
This matters for interpretation: a blocked target is a configuration problem on your side, and it is never reported as a vendor outage. The policy is not relaxed for convenience, including in development.
5. Nine states, because “no data” is ambiguous
An empty chart cannot tell you whether the vendor is down or whether we never ran the probe. Those are different facts with different owners, so each dependency carries an explicit state:
- Target problems - the probe reached the vendor and it failed, or was refused by the security policy.
- Infrastructure problems - dispatch failed, or the scheduler was not proven alive, so no probe could have run.
- Transitional - never checked, awaiting schedule, queued, or executing.
A timeout, a policy block, a worker outage and a dead scheduler are four distinct states, because they have four different causes and four different fixes.
Evidence & data
- Each stored result records its region, outcome, status code, latency and execution time.
- Dispatch failures are counted and logged with the dependency, region and failure reason, and the next scheduled attempt is left untouched so the check is retried rather than silently forgiven.
- Scheduler and worker liveness are published as time-limited heartbeats and surfaced through a health endpoint, so a pipeline that has stopped is reported as broken rather than presenting an empty history.
Methodology & sources
Checks are HTTP requests made by RELIASTRA’s own workers on a fixed interval. There is no sampling frame and no human annotation. Outcomes are determined by the response (or the absence of one) against the expectation configured for that dependency.
Known limitations, stated plainly: our origins are a fixed, small set of regions, so we cannot distinguish a vendor-wide outage from one affecting a geography we do not observe from. We do not measure end-user experience, only server-to-server responses. And a dependency that requires authenticated access is checked with credentials you supply, which means a credential rotation can present as a target failure.
Related research
- The Dependency Gap
Why vendor and self-inflicted outages look identical.
- The RELIASTRA research agenda
Standards we hold our own data to.
Independent research
Follow independent infrastructure research
If you find RELIASTRA’s independent infrastructure intelligence useful, add us as a Preferred Source - you’ll see future incident analysis and dependency research more often when you search.
Powered by Google - you choose your sources.