Multi-instance scenario

Seven steps on the cluster mode of the demo, with what each one shows. Step 6 cuts the S3 store, step 7 goes through Keycloak: deploy with them.

demo/k3s/demo.sh cluster-up
demo/k3s/demo.sh deploy cluster --with-s3 --with-oidc   # or --preset example
demo/k3s/demo.sh check cluster                # the seven steps, checked

Before the steps

. demo/k3s/.work/cluster/env.sh
k get pods -o wide                     # kubectl, run in the k3s container
set -- $(k get pods -l app=craft-file-gate -o jsonpath='{.items[*].metadata.name}')
PA=$1 PB=$2 PC=$3
A=$(pod_ip $PA) B=$(pod_ip $PB) C=$(pod_ip $PC)
TOKEN=$(login $A)

A, B and C are the pods’ addresses. env.sh gives, through Docker alone and with no secret on a command line: k (kubectl), t (the tools container), jq, login <ip> (a console token of admin), adm <ip> <path> (the admin API with $TOKEN), bearer <token> <ip> <path>, files <ip> <path> (the file API as demo), demo_sftp; with Keycloak, oidc_token <alice|bob|carol>, jwt_part <token> 0|1 (header, claims), jwt_files <token> <ip> <path>, jwt_sftp <token>, kc_admin <path> (Keycloak’s admin API).

1. SFTP and the explorer on different pods

sftp -P 32122 demo@127.0.0.1           # $USER_PASSWORD; keep it open (or demo_sftp)
sftp> ls /                             # archive  home  (home alone without S3)
sftp> get /home/welcome.txt

Open the explorer as demo and the console as admin. In the console, Sessions shows the SFTP session and its Instance; API clients shows demo and the pod of the explorer. The node port spreads connections over the pods: when both land on one pod, reconnect.

adm $A '/admin/sessions?scope=cluster' | jq -r '.sessions[] | "\(.username) \(.type) \(.pod_name)"'
adm $A '/admin/api-clients?scope=cluster' | jq -r '.clients[] | "\(.username) \(.instance)"'

2. A ban on one pod, seen by the others

Three wrong passwords on pod A ([api.ban] max_failures = 3), then the right one on B:

for i in 1 2 3; do files $A '?list' -u demo:wrong -o /dev/null -w '%{http_code}\n'; done
files $B '?list'
adm $C '/admin/bans?scope=cluster' | jq -c '.bans[]'
k get configmap gate-bans -o jsonpath='{.data.bans}'

B and C answer 403 IP temporarily banned, within a few hundred milliseconds: the ban is in the gate-bans ConfigMap, which every pod watches. The address is the node’s (10.42.0.1), that of the explorer on the host as well: it is refused too. Bans lists the ban on each instance; Unban on one pod lifts it everywhere:

adm $C /admin/bans/api/10.42.0.1 -X DELETE

3. A console session accepted by all, a logout revoked everywhere

S=$(login $A)
for ip in $A $B $C; do bearer $S $ip /admin/me -o /dev/null -w "$ip %{http_code}\n"; done
bearer $S $B /admin/logout -X POST
for ip in $A $B $C; do bearer $S $ip /admin/me -o /dev/null -w "$ip %{http_code}\n"; done
TOKEN=$(login $A)

The token from A opens B and C (200): the key that signs it is in the shared Secret gate-cluster. After the logout on B, all three answer 401 within a second: the revocation is in gate-access. Logout in the console does the same.

4. Temporary access, visible on all pods

In the console, Temporary access, New access: demo, role drop-zone, 1 h, a reason. Or:

until=$(( $(date +%s) + 3600 ))
GRANT=$(adm $A /admin/grants -X POST -H 'content-type: application/json' \
  -d "{\"username\":\"demo\",\"role\":\"drop-zone\",\"until\":$until,\"reason\":\"demo delivery\"}" | jq -r .grant.id)
adm $B /admin/grants | jq -c '.grants[] | {username, role, state}'
files $C '?list' | jq -r '.entries[].name'

B and C list the access (active); /drop appears to demo on every pod, in the explorer after a reload. A revocation on any pod removes it everywhere:

adm $B /admin/grants/$GRANT -X DELETE

5. The cluster view while a pod is deleted

peers() { adm $A /admin/instances | jq -r '.instances[] | "\(.status) \(.info.pod_name // .instance_id)"'; }
peers                                  # self, reachable, reachable
k delete pod $PC
peers                                  # the deleted pod: unreachable
k rollout status deploy/gate
peers                                  # its replacement: reachable

The Instances tab shows the same: the deleted pod unreachable, then gone when the replacement is ready, three instances again. The console comes back after one failed reload: its port-forward restarts on another pod. Take the new pod for step 6: PC=<its name>; C=$(pod_ip $PC).

6. Storage cut: the detectors and the withdrawal

With S3 only. RustFS cut from one pod alone: a NetworkPolicy refuses it the address of RustFS, and the S3 client’s open connection is dropped.

S3IP=$(k get pod -l app=s3 -o jsonpath='{.items[0].status.podIP}')
k label pod $PC demo-cut=s3
k apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: demo-cut-s3 }
spec:
  podSelector: { matchLabels: { demo-cut: s3 } }
  policyTypes: [Egress]
  egress: [{ to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [$S3IP/32] } }] }]
EOF
docker exec $K3S conntrack -D -s $C -p tcp --dport 9000
k logs -f $PC | grep -E 'backend unavailable|cluster detector|withdrawn from service|back in service'   # Ctrl-C
k get pods -w                          # Ctrl-C

k delete networkpolicy demo-cut-s3 && k label pod $PC demo-cut-

backend available again, back in service 5 s later, 1/1 at the next probe. Then RustFS stopped for everyone:

k scale deploy/s3 --replicas=0
t curl -s http://$A:8081/metrics | grep 'craftfilegate_backend_up\|craftfilegate_unready_detector'
k get pods
k scale deploy/s3 --replicas=1

Every pod logs backend unavailable and publishes craftfilegate_backend_up{backend="archive"} 0; no pod leaves the service: storage_alone needs a peer that sees the storage up, and isolated_and_storage_down an isolated pod. /home stays served, /archive answers 500 until RustFS is back. These detectors: Several instances.

7. The tokens of an OIDC provider

With Keycloak only (The OIDC provider). A token of the demo realm, by the password grant, at each door:

T=$(oidc_token alice)
jwt_part $T 1 | jq '{iss, aud, preferred_username, realm_access}'
jwt_sftp $T                            # sftp -P 32122 jwt@127.0.0.1, the token as password
jwt_files $T $A '?list' | jq -r '.entries[].name'
bearer $T $A /admin/me | jq '{username, auth_method, roles}'
bearer $(oidc_token bob) $A /admin/me -o /dev/null -w '%{http_code}\n'   # 403

alice opens SFTP, the file API and the console (console-admins); bob, without an admin role, the files only. In the explorer (Use a token instead) and the console (Sign in with a token), paste $T.

A rotation: in Keycloak (Realm settings, Keys, Add provider, ecdsa-generated, a priority above the others), or:

TOP=$(kc_admin '/realms/demo/components?type=org.keycloak.keys.KeyProvider' | jq '[.[].config.priority[0] | tonumber] | max')
RID=$(kc_admin /realms/demo | jq -r .id)
kc_admin /realms/demo/components -X POST -H 'content-type: application/json' -d "{\"name\":\"rotated\",
  \"providerId\":\"ecdsa-generated\",\"providerType\":\"org.keycloak.keys.KeyProvider\",
  \"parentId\":\"$RID\",\"config\":{\"priority\":[\"$((TOP + 100))\"]}}"
T=$(oidc_token alice); jwt_part $T 0 | jq -r .kid    # the new key
for ip in $A $B $C; do jwt_files $T $ip '?list' -o /dev/null -w "$ip %{http_code}\n"; done
kc_admin /realms/demo/components/$(kc_admin '/realms/demo/components?name=rotated' | jq -r '.[0].id') -X DELETE

Each pod answers 200 without a restart: a kid it does not know makes it reload the key set (JWKS cache refreshed, trigger unknown_kid), once a minute at most; the DELETE gives the signing back to the old key, which the pods keep. Then carol disabled (Users, carol, Enabled): Keycloak gives her no more tokens (Account disabled), but the one she holds stays valid until its exp, 10 minutes at most: a JWT is not revoked.

With S3: every operation

check.sh also goes through /archive by the file API and by SFTP: upload, list, download, rename and delete, a 9 MiB file (above the 8 MiB multipart threshold) both ways, each object seen in the bucket. The RustFS console (http://127.0.0.1:32190/rustfs/console/, the keys printed by deploy) shows them arrive under archive/gate/.

The single mode

demo/k3s/demo.sh deploy single (ports 32222, 32280, 32281) plays the same steps on one pod: the ban is kept in its file (/state/api-bans.json); the console token stays valid after the pod is replaced, its key being in the shared Secret; with S3 stopped the pod stays in service, without peers there is no detector. demo/k3s/demo.sh check single checks them.