Demo chaos testing
Faults injected into a deployed mode of the demo, one at a time, each with what it must show: what holds, by which mechanism of the server, and how fast.
demo/k3s/demo.sh chaos cluster # every experiment that applies
demo/k3s/demo.sh chaos cluster s3-one # one
demo/k3s/demo.sh pods cluster # each pod's state, as JSON
The same from the Chaos tab of the control page, which also shows the pods’ live state and one card per experiment.
How an experiment runs
| Step | What |
|---|---|
| prerequisites | the mode, the number of pods, the options (--with-s3, --with-oidc) the experiment needs; otherwise SKIP, with the reason |
| initial state | every pod ready and /readyz 200, the mode’s number of replicas, backends up, in cluster mode every peer reachable and no detector active, no Deployment rolling, no NetworkPolicy of an experiment; otherwise refused (REFUSED), and all stops |
| injection | one fault, on the timeline; with --with-otel, a Grafana annotation (tags cfg-demo, chaos) |
| signals | what the hypothesis awaits, each with its delay measured from the injection and a deadline: log lines (kubectl logs -f), /admin/events events, a ConfigMap watch, kubectl wait, metrics, /readyz |
| return | the fault removed (an annotation), then the return’s signals, their delay measured from it |
| restore | what the experiment changed, undone whatever happens (end, error, Ctrl-C); then the initial state awaited again, 300 s at most |
Verdict: OK when every signal came in time, KO otherwise, ERROR when
the experiment stopped before its last signal. Each run:
demo/k3s/.work/<mode>/chaos/<date>-<experiment>.json (hypothesis,
mechanism, rules, timestamped timeline, verdict, delay_s, recovery_s,
restored, Grafana link). The waits are signals; where nothing tells (a
kube-proxy rule, the end of a 60 s delay of the server, a peer’s state read
every second), a short bounded poll, said in demo/k3s/chaos.sh. The
experiments are described in demo/k3s/chaos.json, which the script and
the page read.
The experiments
| Experiment | Prerequisites | Injection | What it shows | Rules |
|---|---|---|---|---|
pod-kill | none | kubectl delete pod of the pod receiving an SFTP upload (6 MiB at 1 MiB/s) | the pod leaves the service at the signal (/readyz 503), the upload ends whole within the grace, graceful shutdown complete; a new session is served by another pod (in single, by the replacement) | R-SHUTDOWN-002, 004, 009 |
isolate | cluster, 3 pods | two egress NetworkPolicies: the last pod reaches no peer, the others do not reach it | the cut pod says it is isolated (detector isolated, craftfilegate_cluster_isolated 1), its peers see it unreachable, nobody leaves the service: unreadyWhen: auto does not name isolated | R-CLUSTER-006, 015, 016 |
s3-one | cluster, --with-s3 | a NetworkPolicy: the last pod no longer reaches RustFS | storage_alone, the withdrawal (/readyz withdrawn_by, out of the gate-api endpoints), still a peer; back after ready_after_secs | R-AVAIL-002, R-CLUSTER-015, 016 |
s3-all | --with-s3 | RustFS at 0 replicas | every pod sees archive down, none withdraws, the local backend serves | R-AVAIL-002, R-CLUSTER-016 |
k8s-api | none | a NetworkPolicy: the server pods no longer reach the API server (the node’s address), their open connections dropped (conntrack) | a temporary access and a ban decided during the cut hold on their pod (local_only), the pods stay ready; once back, the access and the ban reach the ConfigMaps and the peers, and so does a ban decided after the return | R-GRANT-005, R-BAN-017 |
keycloak | --with-oidc | a NetworkPolicy: the pods no longer reach Keycloak, which keeps signing; a new key in the realm | the cached JWKS keys verify the tokens already signed; a token of the new key is refused during the cut, accepted once back without a restart | R-AUTH-026, 027 |
auth-burst | none | wrong passwords on the file API, one pod after the other, until the first 403 | one ban, held by every pod (ban.added), lifted everywhere; in single, after max_failures failures at most (those still in the window count) | R-BAN-004, R-CLUSTER-011, R-BAN-017 |
hash-pool | none | the pods restarted with CRAFT_FILE_GATE_HASH_QUEUE=1 (2 places), then sign-ins of demo, 4 at a time, on one pod, while the probes run | 503 password checks saturated with Retry-After: 1, craftfilegate_password_hash_pool_full_total going up, while /livez, /readyz and the console answer | R-AUTH-011, R-K8S-002 |
The limits
| Experiment | Limit |
|---|---|
| all | one node: a cut is a NetworkPolicy, not a lost link; the node (the kubelet, the tools container, kubectl) still reaches every pod |
pod-kill | an upload longer than shutdown_grace_period_secs is cut; kube-proxy drops the pod from its rules on its own time, the new session is retried for 10 s |
isolate, s3-one | the delays follow the demo’s configuration (probe 5 s, unready_after_secs 10, ready_after_secs 5); with the defaults, a withdrawal takes about 2 min |
s3-all | the pods see the outage a few seconds apart: storage_alone may light up for a moment on one of them, without a withdrawal (the timeline counts those lines) |
k8s-api | in single, the bans live in a file: the access alone is tried |
keycloak | after a failed fetch, an unknown kid waits 60 s: the new token is asked again every 2 s, 75 s at most |
auth-burst | a peer’s state is read every second: a faster burst passes up to max_failures failures per pod before the sum catches up (the timeline gives the attempts and peer_failures) |
hash-pool | one address holds 4 places at most (hash_per_address): the pool is shrunk so that one address fills it; two rolling updates, one at the injection, one at the return |