consolidate VELA mqtt values to the external branch; document the precedence trap

The backend configmap resolves the broker as enabled → external.host →
monitoring.releaseName, but the frontend nginx /mqtt proxy checks
monitoring.releaseName before external.host. VELA-values set both, so
edits to external.wsPort/useTls never reached nginx, and clearing
releaseName exposed external.wsPort: 443 — a dead port on the in-cluster
service that fails the TLS handshake. Now only external.* is set, with
wsPort 9001 (the broker's actual WebSocket listener).

Debugging doc gains a "Values gotchas" section (single-branch rule,
wsPort 9001 vs 443, no rewrites on /mqtt — WebSocket clients don't
follow redirects) and matching failure-table rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chart/mgmt-suite/VELA-values.yaml

@@ -70,15 +70,19 @@ postgresql:
 
 mosquitto:
   enabled: false
+  # Exactly ONE of monitoring.releaseName / external.host may be set: the backend
+  # config prefers external.*, but the frontend nginx /mqtt proxy prefers
+  # monitoring.* — with both set they resolve the broker independently and can
+  # disagree. We use external.* only; see docs/mqtt-debugging.md §1 gotchas.
   monitoring:
-    releaseName: "mgmt-monitoring"
+    releaseName: ""
     useTls: true
   external:
-    host: "mgmt-monitoring-mosquitto.monitoring.svc.cluster.local"              # e.g., "mqtt.example.com" or "a1b2c3.iot.us-east-1.amazonaws.com"
-    port: 8883            # External brokers typically use 8883 (TLS)
+    host: "mgmt-monitoring-mosquitto.monitoring.svc.cluster.local"
+    port: 8883            # Broker MQTT-over-TLS listener (backend publish connection)
     wsUrl: ""             # Empty → derives wss://<ingress.host>/mqtt (manage.k8s.osa.n.com), the host the ACM/F5 cert covers. Do NOT point at a separate mqtt.* host — its SAN won't match and TLS fails.
-    wsPort: 443           # WebSocket port for browser clients
-    useTls: true
+    wsPort: 9001          # Broker's in-cluster WebSocket listener. 443 is only for external brokers behind their own LB — against the k8s service it's a dead port.
+    useTls: true          # TLS on 8883 (backend) AND on the nginx /mqtt → 9001 hop
 
 # Credentials should be in the existingSecret as JIRA_USERNAME and JIRA_PASSWORD.
 jira:

docs/mqtt-debugging.md

@@ -33,6 +33,26 @@ Browser-facing WebSocket URLs (browsers can never reach 8883/9001 directly — o
 | `wss://mqtt.k8s.osa.n.com/mqtt` | dedicated ALB ingress (monitoring chart) → HTTPS straight to broker's TLS 9001 | Internal-scheme ALB: resolvable/reachable only inside the VPC. Backend protocol is HTTPS, so this is the correct path when broker TLS is on. |
 | `ws://localhost:9001/mqtt` | direct to local mosquitto | dev/demo only |
 
+### Values gotchas (read before changing `mosquitto.*`)
+
+- **Set exactly one of `monitoring.releaseName` / `external.host`.** The backend
+  configmap resolves the broker as `enabled` → `external.host` → `monitoring.releaseName`,
+  but the frontend nginx `/mqtt` proxy resolves it as `enabled` → `monitoring.releaseName`
+  → `external.host`. With both set, the backend follows `external.*` while nginx follows
+  `monitoring.*` — editing `external.wsPort`/`external.useTls` then has no effect on the
+  browser path, and the two can silently point at different brokers.
+- **`external.wsPort` must be the broker's real WebSocket listener.** For the in-cluster
+  monitoring broker service that is `9001`. `443` is only correct for a genuinely
+  external broker behind its own load balancer — against the k8s service it makes nginx
+  do a TLS handshake on a port nothing listens on (`SSL_do_handshake() failed` /
+  "unexpected EOF"-style errors).
+- **Don't add rewrites for `/mqtt/*` in nginx.** The app connects to the literal path
+  `/mqtt` (`ws_url` = `wss://<host>/mqtt`; mqtt.js appends nothing), and mosquitto's
+  WebSocket listener ignores the HTTP path entirely — a bare
+  `location /mqtt { proxy_pass ...; }` is sufficient. A `rewrite` that returns a
+  redirect (30x) breaks the connection outright: browser WebSocket clients don't follow
+  redirects.
+
 Topics in use:
 
 - `mgmt/announcements/alert`, `mgmt/announcements/bulletin` — backend → browsers
@@ -447,3 +467,6 @@ Broker log lines (`kubectl logs deploy/mgmt-monitoring-mosquitto -n monitoring`)
 | WS connects, drops every N seconds | LB idle timeout below MQTT keepalive | ALB `idle_timeout` is set to 3600s in VELA values; mirror on the F5 VIP; nginx already holds 86400s |
 | `mqtt-status` shows DNS ok, TCP fail | NetworkPolicy / security group between namespaces or VPC → broker | Check policies on the `monitoring` namespace and the node SGs |
 | Works via port-forward, fails via `wss://mqtt.*` | Ingress/ALB layer (cert-arn, subnets, internal scheme, healthcheck failing so no targets) | `kubectl describe ingress mgmt-monitoring-mosquitto -n monitoring`; ALB target health |
+| nginx log: `SSL_do_handshake() failed` or `connect() failed (...:443)` on the `/mqtt` hop | nginx proxying to `external.wsPort: 443` against the in-cluster broker service (only 8883/9001 exist), or TLS upstream to a plaintext listener | Set `external.wsPort: 9001`; make `useTls` match the listener (§1 gotchas) |
+| Changed `external.*` values but browser behavior didn't change | `monitoring.releaseName` still set — it outranks `external.*` inside the nginx `/mqtt` proxy (opposite of the backend's order) | Blank out `monitoring.releaseName` (or use only `monitoring.*`); see §1 gotchas |
+| WS request gets a 301/302 instead of 101 | A `rewrite`/redirect added to the nginx `/mqtt` location | Remove it — WebSocket clients don't follow redirects, and mosquitto ignores the path anyway |