A Planet On A Laptop

2026-11-04来源 dev.toEnglish
A Planet On A Laptop

A Planet On A Laptop - the cover: two continents, providers and datacenters drawn as nested rings, measured round trip close to declared

A Planet On A Laptop

Continents, providers, datacenters and zones declared in one file, turned into netem and iptables rules inside every container, broken at every level on purpose - and a measured round trip that came out close to what was declared.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The topology emulation is built and has unit tests; placement and links were applied on a laptop and a round trip was measured. The failure scenarios on top of it are written and not run - the owner postponed them.

This is part 3 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

One file

Part 2, Four Failure Domains, ended with a ladder: zone, datacenter, provider, continent. This part puts that ladder on a laptop.

Everything starts from one declaration file. It is switched on by an environment variable, and only in the high-availability profile - a normal developer environment does not pay for it. The form, with cities and round trips as an example:

continents: eu: providers: prov-a: dc-a1: { city: Frankfurt, region: eu-frankfurt, zones: [dc-a1-z1, dc-a1-z2] } dc-a2: { city: Amsterdam, … } prov-b: dc-b1: { city: Paris, … } us: providers: prov-b: dc-b2: { city: Virginia, … } links: intra_dc: { rtt_ms: 0.5, jitter: 0.1, loss: 0 } pairs: dc-a1~dc-a2: { rtt_ms: 11, jitter: 1, loss: 0 } dc-a1~dc-b2: { rtt_ms: 87, jitter: 3, loss: 0.1 } clients: client-eu: { continent: eu, dc: dc-a1, rtt_ms: 20, jitter: 2 } placement: {} # override <service|cluster>: [zone, …]

Two continent rings side by side; inside the left one two providers, inside them datacenters, inside those zone dots; the right ring holds one datacenter of the second provider; links drawn between datacenters

A dark card with the declaration: continents, providers, datacenters with a city and zones, links with rtt, jitter and loss, a client, an empty placement override; the transatlantic link highlighted

Where the numbers come from

The round trips in the file are not guesses. Each is the median of public ping data between the two cities, cross-checked against a second public source. The declaration says what the internet between those cities looks like, not what we wish it did.

The kind of every link is derived from where its ends are, in the order from part 2: intra-dc, intra-metro, intra-continent, inter-continent. Continent ranks above provider; provider is only same or cross.

Placement is deterministic: copies are dealt round-robin over providers, then datacenters, then zones. Three copies land in three datacenters; a fourth goes to the other continent. Each copy is told where it is through four variables - APP_PROVIDER, APP_REGION, APP_SITE, APP_ZONE - and part 4 is about what happens to them next.

From a declaration to rules

The declaration becomes a model, and the model becomes rules inside every container.

A flow: declaration, model, a sidecar per container, rules in the container's namespace; a pill reads built, unit tests

The rules are applied from a sidecar that joins the container's network namespace - so no service image needs tc, iptables or extra capabilities:

docker run --rm --net container:<name> --cap-add NET_ADMIN nicolaka/netshoot \ sh -c ' tc qdisc add dev eth0 root handle 1: prio bands <one per peer> tc qdisc add dev eth0 parent 1:<band> netem delay <rtt/2> <jitter> loss <loss> tc filter add dev eth0 parent 1: protocol ip u32 match ip dst <peer subnet> flowid 1:<band> '

A prio qdisc gives one band per peer; netem in the band adds half the round trip, the jitter and the loss; a u32 filter sends traffic for the peer's subnet into its band. The mirror rule on the other side adds the other half, so the round trip equals the declared one. The netem syntax is the kernel's own (tc-netem(8)); the sidecar image is netshoot.

A dark terminal card with the sidecar command and three tc lines - prio, netem delay half the rtt with jitter and loss, a u32 filter on the peer subnet - and one iptables DROP line for a cut

The rules are re-applied on every up, start and restart, because a recreated container comes back with a fresh namespace and none of them.

A cut is a DROP

A link cut is not a stopped container. It is a rule on both sides:

iptables -I OUTPUT -d <peer subnet> -j DROP

Both ends stay alive and both believe they are fine; only the path between them is gone. That is the case a stopped container never shows.

Compose: one network, many subnets

In compose, the whole planet is one docker network. Each datacenter gets its own /20, each zone its own /24, and every container a static address in a generated overlay.

One wide network tile; inside it datacenter blocks each labelled /20, inside each two zone blocks labelled /24 with container dots; a crossed-out alternative reads a network per datacenter - docker DNS is per network

A network per datacenter looked cleaner and was rejected: docker's DNS is per network, so service names would stop resolving across datacenters. One network with subnets keeps every name working and still gives every link its own address range to match on.

k3d: a node per zone

In k3d, each zone is one agent node, labelled with topology.kubernetes.io/region and topology.kubernetes.io/zone plus provider and datacenter labels. The spread is two constraints with different strictness:

topologySpreadConstraints: - { maxSkew: 1, topologyKey: <datacenter label>, whenUnsatisfiable: DoNotSchedule } - { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway }

Across datacenters the scheduler must spread; across zones it should (Kubernetes). The same netem and iptables rules run inside the node containers.

Why not a real VPN

In production the links between datacenters are a VPN. WireGuard was considered and rejected: it would need NET_ADMIN and route changes in every service image. The emulation gives the links the latency and loss of a VPN without touching a single service.

Breaking it at every level

Every level of the ladder has its own action:

Action What it does
zone-down / dc-down / provider-down and -up stop and start the members; rules re-applied on start
dc-isolate / dc-heal DROP from one datacenter to every other
continent-isolate / continent-heal the same, for a continent
link-down / link-up cut one pair
link-degrade tc qdisc change … netem with a given rtt, loss and jitter
link-restore back to the declared values

Five icon tiles, one per level - zone, datacenter, provider, continent, link - each naming its actions

A scenario can aim at a role, instance: leader, rather than at a name. Every run records which latency variant it used, --latency none or --latency declared, so a report always says whether the network was there.

A cut remembers why

Cuts overlap. A link can be cut on its own, and a datacenter at one end can be isolated on top. Healing the datacenter must not quietly heal the link.

So every cut remembers its cause - link, datacenter or continent - and healing removes only the cuts of that cause. The state lives in a JSON file per environment.

An animation: a link is cut, cause link; then the datacenter is isolated, cause dc; then the datacenter is healed and its cuts disappear, but the link cut stays red

Measured

Placement and links were applied on a laptop. A ping across a declared 11 ms link came out slightly above what was declared: the minimum almost exact, the average a little higher. Inside one zone, near zero.

Two halves: on the left declared - the link as written in the file; on the right measured - minimum almost exact, average slightly above; same zone near zero

The earlier store failover runs of Seven Ways To Break It On Purpose were done without any of this latency. That is worth saying out loud: their results hold for a flat network, and the next runs are the first with distance in them.

Written, not run

The scenarios are written. None has run - the owner postponed them:

  • a zone down under load, in compose and in k3d;
  • the zone of a database leader down;
  • a broker's or a cache's datacenter isolated;
  • a provider down;
  • one link down; one link degraded to a high round trip with loss; the transatlantic link degraded;
  • a continent cut off; an even split.

A flow of dashed ghost tiles: zone down, leader zone, dc isolate, provider down, link degrade, continent cut, even split; a pill reads planned, the owner postponed the runs

QA has declared what a pass would mean. It is a proposal, agreed with DevOps and postponed by the owner, and these numbers live only here, in the labelled block:

# HA thresholds - a proposal, agreed with DevOps, postponed by the owner rtt_per_declared_link: ±20 % leader_election: ≤ 35 s replication_lag: { within_continent: ≤ 1 s, across: ≤ RTT×3 + 1 s } rejoin: ≤ 60 s broker_elections: recorded only split_brain: always ❌

They are declared, not met.

Not emulated

Bandwidth limits, BGP and anycast, a provider's DNS failing, clock skew between datacenters, and how operator-managed StatefulSets spread in k3d. Each is a known gap, not a forgotten one.

What I do not claim

  • Not that any failure scenario has run on this topology. They are written and planned.
  • Not that the thresholds are met. They are a proposal.
  • Not that a laptop network is a planet. The measured round trip came close to declared, slightly above, and that is all that was measured.
  • Not bandwidth, BGP, provider DNS or clock skew. They are not emulated.

What of this is method

  • Declare the planet once. Rules, placement and chaos all read one file.
  • Rules in the namespace, not in the image. A sidecar keeps every service untouched.
  • Cuts remember their cause. Healing one failure must not hide another.

Cross-anything on a laptop - Part 3. Next: Placement In Every Trace.

If you emulate latency with netem, do you set it per peer or per container? If you use a chaos tool for partitions, how does it heal overlapping cuts? If you run k3d with zone labels, what spreads and what does not? What is the first link on your planet you would cut?

原文：https://www.a8q9.cn/en/article/a-planet-on-a-laptop-f93d13.html

相关文章