Demo on k3s

A local Kubernetes in Docker, on which the server is deployed one way or another: one pod, three pods that share their state, or the same objects without the chart. Docker is the only prerequisite.

demo/k3s/demo.sh cluster-up                  # k3s in Docker; again: resumes and repairs
demo/k3s/demo.sh deploy cluster              # the chart, 3 replicas
demo/k3s/demo.sh check cluster               # plays the scenario and checks it
demo/k3s/demo.sh deploy single --preset example --with-s3
demo/k3s/demo.sh deploy cluster --with-oidc  # with Keycloak, an OIDC provider
demo/k3s/demo.sh list                        # modes, deployed or not, ports
demo/k3s/demo.sh remove single               # its namespace only
demo/k3s/demo.sh cluster-down                # every cfg-demo-* container, volume, network

deploy prints the addresses and the accounts, with passwords drawn at each deployment and hashed by the image’s hash-password; they stay in demo/k3s/.work/<mode>/env.sh. The steps: Multi-instance scenario. After an interrupted run or a docker restart cfg-demo-k3s, cluster-up waits for the API server again, recreates the tools container and restarts the console port-forwards of the deployed modes.

Modes

A mode is the topology: demo/k3s/modes/<mode>/. Each lives in its namespace cfg-demo-<mode>, its release is called gate, and the modes coexist on their own host ports.

The explorer is http://127.0.0.1:<port>/files/, the console http://127.0.0.1:<port>/ (a kubectl port-forward to gate-control), the S3 console http://127.0.0.1:<port>/rustfs/console/, Keycloak’s http://127.0.0.1:<port>/admin/. CFG_DEMO_PORT_BASE (32000) moves every port, CFG_DEMO_BIND (127.0.0.1) the address; both are read by cluster-up.

Another mode is another directory: a slot not taken (1 to 4), its values.

Presets

A preset is the content, the same in every mode: demo/k3s/presets/<name>/, or any directory (--preset DIR, or PRESET=DIR). Every file is optional; without the file, presets/default/ serves.

The mode adds what is its own, in every configuration: backend (ConfigMap) or persist_file (file) under each [sftp.ban], [api.ban], [admin.ban], [cluster], and the JWT method (jwt under [auth.methods], off, and [auth.jwt] with --with-oidc); the chart passes the shared Secret, the revocations, the grants and the peers to the server. A preset’s config.toml leaves out [cluster], [auth.jwt], auth.methods.jwt and, under the ban, session and grants tables, backend, persist_file, key_file, secret_name and the ConfigMap names. Before anything is deployed, the configuration goes through config check of the image, with the mode’s variables, and the values through helm template.

presets/example/ uses every file: fixed passwords, an account partner, the DEMO environment in green, a file and an object, a ban, an access, S3.

The S3 store

Without S3, the local backend alone: enough to drop and fetch files, and the lightest. --with-s3 adds RustFS (rustfs/rustfs:1.0.3, pulled by k3s on first use) and the archive backend (/archive): its keys, drawn at deployment, are printed with the accounts, and its web console shows the objects as they arrive. Measured here: a server pod holds about 5 MiB, RustFS about 60 MiB.

The OIDC provider

--with-oidc adds Keycloak (quay.io/keycloak/keycloak:26.8.1, pulled by k3s, in start-dev: its database lives and dies with the pod) and the demo realm, imported at startup from demo/k3s/oidc-realm.json. The server starts after it: its startup loads the keys.

The passwords of the three, and that of admin, Keycloak’s administrator, are drawn at deployment and printed with the others. The mode adds to the configuration (without --with-oidc, jwt = { enabled = false }):

[auth.methods]
jwt = { enabled = true }

[auth.jwt]
jwks_url = "http://keycloak:8080/realms/demo/protocol/openid-connect/certs"
issuer = "http://127.0.0.1:32191/realms/demo"     # the mode's Keycloak port
audience = "craft-file-gate"
username_path = "/preferred_username"
authorities_path = "/realm_access/roles"

KC_HOSTNAME fixes the address Keycloak writes into iss: a token taken on the host, in the tools container or in a pod names the same issuer. Its host is CFG_DEMO_PUBLIC_HOST, else CFG_DEMO_BIND (127.0.0.1 when it is 0.0.0.0). The realm lets no one register or change their username: preferred_username can serve as the name. The keys come over http inside the cluster, and config check as well as the startup say so in a WARN: acceptable for a demo, not in production. After a restart of k3s, cluster-up restarts Keycloak (the realm imported again, a new key, the same passwords), then the server. Measured here: Keycloak ready in about 20 s, about 600 MiB (1 GiB limit).

What is built

Short delays, for the demo, in presets/default/config.toml:

[server.backend_probe]
interval_secs = 5           # a storage outage is seen in about 7 s
timeout_secs = 2
failures_before_down = 1

[admin.session]
reread_interval_secs = 1    # likewise [admin.grants]

The cluster mode adds unready_after_secs = 10 and ready_after_secs = 5 under [cluster].