Multi-instance scenario

Six steps on the cluster mode of the demo, with what each one shows. Step 6 cuts the S3 store: deploy with it.

demo/k3s/demo.sh cluster-up
demo/k3s/demo.sh deploy cluster --with-s3     # or --preset example
demo/k3s/demo.sh check cluster                # the six steps, checked

Before the steps

. demo/k3s/.work/cluster/env.sh
k get pods -o wide                     # kubectl, run in the k3s container
set -- $(k get pods -l app=craft-file-gate -o jsonpath='{.items[*].metadata.name}')
PA=$1 PB=$2 PC=$3
A=$(pod_ip $PA) B=$(pod_ip $PB) C=$(pod_ip $PC)
TOKEN=$(login $A)

A, B and C are the pods’ addresses. env.sh gives, through Docker alone and with no secret on a command line: k (kubectl), t (the tools container), jq, login <ip> (a console token of admin), adm <ip> <path> (the admin API with $TOKEN), bearer <token> <ip> <path>, files <ip> <path> (the file API as demo), demo_sftp.

1. SFTP and the explorer on different pods

sftp -P 32122 demo@127.0.0.1           # $USER_PASSWORD; keep it open (or demo_sftp)
sftp> ls /                             # archive  home  (home alone without S3)
sftp> get /home/welcome.txt

Open the explorer as demo and the console as admin. In the console, Sessions shows the SFTP session and its Instance; API clients shows demo and the pod of the explorer. The node port spreads connections over the pods: when both land on one pod, reconnect.

adm $A '/admin/sessions?scope=cluster' | jq -r '.sessions[] | "\(.username) \(.type) \(.pod_name)"'
adm $A '/admin/api-clients?scope=cluster' | jq -r '.clients[] | "\(.username) \(.instance)"'

2. A ban on one pod, seen by the others

Three wrong passwords on pod A ([api.ban] max_failures = 3), then the right one on B:

for i in 1 2 3; do files $A '?list' -u demo:wrong -o /dev/null -w '%{http_code}\n'; done
files $B '?list'
adm $C '/admin/bans?scope=cluster' | jq -c '.bans[]'
k get configmap gate-bans -o jsonpath='{.data.bans}'

B and C answer 403 IP temporarily banned, within a few hundred milliseconds: the ban is in the gate-bans ConfigMap, which every pod watches. The address is the node’s (10.42.0.1), that of the explorer on the host as well: it is refused too. Bans lists the ban on each instance; Unban on one pod lifts it everywhere:

adm $C /admin/bans/api/10.42.0.1 -X DELETE

3. A console session accepted by all, a logout revoked everywhere

S=$(login $A)
for ip in $A $B $C; do bearer $S $ip /admin/me -o /dev/null -w "$ip %{http_code}\n"; done
bearer $S $B /admin/logout -X POST
for ip in $A $B $C; do bearer $S $ip /admin/me -o /dev/null -w "$ip %{http_code}\n"; done
TOKEN=$(login $A)

The token from A opens B and C (200): the key that signs it is in the shared Secret gate-cluster. After the logout on B, all three answer 401 within a second: the revocation is in gate-access. Logout in the console does the same.

4. Temporary access, visible on all pods

In the console, Temporary access, New access: demo, role drop-zone, 1 h, a reason. Or:

until=$(( $(date +%s) + 3600 ))
GRANT=$(adm $A /admin/grants -X POST -H 'content-type: application/json' \
  -d "{\"username\":\"demo\",\"role\":\"drop-zone\",\"until\":$until,\"reason\":\"demo delivery\"}" | jq -r .grant.id)
adm $B /admin/grants | jq -c '.grants[] | {username, role, state}'
files $C '?list' | jq -r '.entries[].name'

B and C list the access (active); /drop appears to demo on every pod, in the explorer after a reload. A revocation on any pod removes it everywhere:

adm $B /admin/grants/$GRANT -X DELETE

5. The cluster view while a pod is deleted

peers() { adm $A /admin/instances | jq -r '.instances[] | "\(.status) \(.info.pod_name // .instance_id)"'; }
peers                                  # self, reachable, reachable
k delete pod $PC
peers                                  # the deleted pod: unreachable
k rollout status deploy/gate
peers                                  # its replacement: reachable

The Instances tab shows the same: the deleted pod unreachable, then gone when the replacement is ready, three instances again. The console comes back after one failed reload: its port-forward restarts on another pod. Take the new pod for step 6: PC=<its name>; C=$(pod_ip $PC).

6. Storage cut: the detectors and the withdrawal

With S3 only. RustFS cut from one pod alone: a NetworkPolicy refuses it the address of RustFS, and the S3 client’s open connection is dropped.

S3IP=$(k get pod -l app=s3 -o jsonpath='{.items[0].status.podIP}')
k label pod $PC demo-cut=s3
k apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: demo-cut-s3 }
spec:
  podSelector: { matchLabels: { demo-cut: s3 } }
  policyTypes: [Egress]
  egress: [{ to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [$S3IP/32] } }] }]
EOF
docker exec $K3S conntrack -D -s $C -p tcp --dport 9000
k logs -f $PC | grep -E 'backend unavailable|cluster detector|withdrawn from service|back in service'   # Ctrl-C
k get pods -w                          # Ctrl-C

k delete networkpolicy demo-cut-s3 && k label pod $PC demo-cut-

backend available again, back in service 5 s later, 1/1 at the next probe. Then RustFS stopped for everyone:

k scale deploy/s3 --replicas=0
t curl -s http://$A:8081/metrics | grep 'craftfilegate_backend_up\|craftfilegate_unready_detector'
k get pods
k scale deploy/s3 --replicas=1

Every pod logs backend unavailable and publishes craftfilegate_backend_up{backend="archive"} 0; no pod leaves the service: storage_alone needs a peer that sees the storage up, and isolated_and_storage_down an isolated pod. /home stays served, /archive answers 500 until RustFS is back. These detectors: Several instances.

With S3: every operation

check.sh also goes through /archive by the file API and by SFTP: upload, list, download, rename and delete, a 9 MiB file (above the 8 MiB multipart threshold) both ways, each object seen in the bucket. The RustFS console (http://127.0.0.1:32190/rustfs/console/, the keys printed by deploy) shows them arrive under archive/gate/.

The single mode

demo/k3s/demo.sh deploy single (ports 32222, 32280, 32281) plays the same steps on one pod: the ban is kept in its file (/state/api-bans.json); the console token stays valid after the pod is replaced, its key being in the shared Secret; with S3 stopped the pod stays in service, without peers there is no detector. demo/k3s/demo.sh check single checks them.