Demo observability

The server’s traces and metrics in Grafana, while the demo runs: --with-otel, compatible with every mode, preset and other option.

demo/k3s/demo.sh deploy cluster --with-otel --with-s3 --with-proxy
demo/k3s/demo.sh check cluster                # step 9 checks traces and metrics
demo/k3s/demo.sh annotate cluster "RustFS stopped" chaos   # a marker on the curves

What is deployed

grafana/otel-lgtm (ghcr.io/grafana/docker-otel-lgtm:v0.37.0, Grafana Labs’ image, pulled by k3s on first use) in the mode’s namespace, one pod: the OpenTelemetry collector, Prometheus, Tempo, Loki and Grafana (demo/k3s/otel.yaml). It starts before the server, which sends it its first spans.

Host port	What
`32130` (`cluster`), `32230` (`single`), `32330` (`ansible`)	Grafana, a `kubectl port-forward` to the Service `otel`

Grafana’s admin account has a password drawn at deployment, printed with the others and kept in env.sh (GRAFANA_PASSWORD); Grafana admits no anonymous access and downloads nothing. Measured here: LGTM ready in 5 to 15 s, 470 to 640 MiB (limit 1 GiB).

What the mode adds

[telemetry]
enabled = true
otlp_endpoint = "http://otel:4317"
protocol = "grpc"
service_name = "craft-file-gate"
metrics = true
metrics_interval_secs = 10    # short, for the demo

[telemetry] belongs to the mode: a preset’s config.toml leaves it out. Each pod gets OTEL_RESOURCE_ATTRIBUTES (service.instance.id and k8s.pod.name: the pod’s name, through the downward API) in the chart’s extraEnv, or the playbook’s extra_env: the pods’ OTLP series stay apart. The mode also sets service.probes.port: 8082 (the playbook’s probes_port): /metrics and the kubelet’s probes on a port of their own, which serves nothing else. The keys: Telemetry.

Signal	Path	In Prometheus or Tempo
spans	OTLP gRPC to the collector, then Tempo	`service.name` `craft-file-gate`, each span names its pod
pushed metrics	OTLP every 10 s	`job="craft-file-gate"`, `instance`: the pod’s name
scraped metrics	Prometheus reads each pod’s `/metrics`, probes port `8082`, every 10 s	`job="craftfilegate"`, `instance`: the pod’s address
span metrics	Tempo, from the spans	`traces_spanmetrics_*`, with exemplars leading to their trace

Prometheus finds the pods through the headless Service otel-scrape (DNS resolution: no RBAC, nothing outside the namespace); it keeps the pods not ready, a pod withdrawn from service by a detector among them. Both sources have the same series names (Metrics).

The network

The server’s NetworkPolicy filters ingress only: nothing holds back the OTLP export. LGTM’s admits, among the pods, the server’s alone, on 4317 and 4318; Grafana, Tempo and Prometheus answer the node only (port-forward, tools container). The NetworkPolicy otel-scrape admits LGTM’s pod (app: otel) on the server pods’ probes port, and there alone; controlFrom stays the mode’s. With --with-proxy, its address is among the trusted_proxies (the node’s pods), but the probes port reads no client address: check verifies that LGTM reads /metrics and reaches neither 2222, 8080 nor 8081.

The dashboard

CraftFileGate (demo/k3s/otel-dashboard.json, provisioned at startup, Grafana’s home page). The variable Source picks the pushed or the scraped metrics.

Row	Panels
Sessions and transfers	SFTP sessions, file API requests, SFTP operations, spans and p95 latency per operation, recent traces
Throughput	SFTP bytes, storage bytes per backend
Refusals and bans	SFTP refusals (limit, PROXY header), ban verdicts, JWT tokens refused, temporary access
Password hashing pool	places in use and places, checks refused pool full
Cluster	peers by status, active detectors, withdrawal from service
Backends	`backend_up`, errors and calls, probe duration
Startup and readiness	pods answering `/metrics`, restarts, memory
Telemetry	spans ended, exported, lost

A panel’s links (icon at the top left) open Explore on Tempo, a TraceQL query of its subject, over the period shown; an exemplar of the latency panel (a diamond) opens its trace.

Annotations

demo.sh annotate <mode> <text> [tag...], or annotate <text> [tag...] after . demo/k3s/.work/<mode>/env.sh, sets a marker (tag cfg-demo, which the dashboard shows) through Grafana’s HTTP API, the password in curl’s configuration, read on stdin. deploy, check (start and result) and remove set one.

Step 9: watching during the scenario

Open the dashboard on the last half hour, then run demo/k3s/demo.sh check cluster:

Step of the scenario	On the curves
1, 2	SFTP sessions and requests; a peak of ban verdicts
5	a pod leaves “Pods answering /metrics”, its replacement appears
6	`storage_alone`, then `Withdrawn` at `1` on a pod; `backend_up` at `0` everywhere while RustFS is stopped
8	refusals and the ban of a client, through Traefik
9	an SFTP upload and a read through the API: their traces in Tempo, the pod in `service.instance.id`

Step 9 checks: the upload’s trace (sftp_operation) and the read’s (api_operation) in Tempo, each pod in Prometheus over OTLP and by the scrape of /metrics (up at 1), the dashboard provisioned and refused without credentials, an annotation set and read back.

Keyboard shortcuts