Inherit nginx x509 header config, add operator runbook
- LookupConfig: ssl-client-cert / ssl-cert-chain-prefix / certificate-chain-length now fall back to the built-in nginx provider's settings (Config scope, then KC_SPI_X509CERT_LOOKUP_NGINX_* env vars), so switching the provider to "hybrid" needs no duplicated Helm values. Explicit hybrid-scope settings still override. - Add LookupConfigTest covering inheritance, override, and defaults (14 tests total, all passing). - Add RUNBOOK.md: phase-by-phase deployment procedure sized to 30-min test windows, with verification checklists, forged-header negative test, rollback per phase, and failure triage table. - README/Dockerfile.example updated to match; stop tracking target/ build output (.gitignore added). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
188
keycloak-hybrid-x509-spi/RUNBOOK.md
Normal file
188
keycloak-hybrid-x509-spi/RUNBOOK.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Runbook: Rolling out the Hybrid X509 Lookup SPI
|
||||
|
||||
Step-by-step deployment procedure for operators. Background, config reference, and
|
||||
troubleshooting tables live in [README.md](README.md) — this document is the "what do I
|
||||
actually type" guide.
|
||||
|
||||
**What we're changing and why (30 seconds):** Today all user traffic passes through the
|
||||
F5 BigIP, which validates user CAC/x509 certs and forwards the cert to Keycloak in an HTTP
|
||||
header. New Transit Gateway (TGW) connectivity from brokerage AWS accounts bypasses the F5,
|
||||
so Keycloak must additionally accept certs **directly from the TLS handshake**, while
|
||||
making sure the F5's cert header **can't be forged** by TGW clients. This SPI does both.
|
||||
|
||||
```
|
||||
Today: user ──mTLS──> F5 (validates cert, adds header) ──TLS──> NLB(L4) ──> Keycloak
|
||||
New TGW path: user ────────────────────mTLS───────────────────────────> NLB(L4) ──> Keycloak
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Roles and coordination
|
||||
|
||||
| Who | Needed for | Task |
|
||||
|---|---|---|
|
||||
| Keycloak/EKS operator | every phase | image build, Helm values, log checks |
|
||||
| F5/BigIP team | Phase 3 | add client cert to the Server SSL profile (see README §6 one-pager) |
|
||||
| TGW-side tester | Phases 2 & 4 | a browser + CAC on a machine that reaches Keycloak *without* the F5 |
|
||||
|
||||
Each phase is designed to fit one ~30-minute test window and to be independently
|
||||
verifiable and reversible. **Do not combine phases** — the log output of each phase is the
|
||||
input to the next.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Prep (no deployment)
|
||||
|
||||
1. Collect from the current deployment:
|
||||
- [ ] Keycloak version → set `<keycloak.version>` in `pom.xml` to match (24.x–26.x).
|
||||
- [ ] Base image tag and whether the entrypoint uses `start --optimized`.
|
||||
- [ ] Truststore config: `KC_TRUSTSTORE_PATHS` (KC 25+) or `spi-truststore-file-*`
|
||||
env vars. The user-CA bundle must be present (it already is if header login works).
|
||||
- [ ] Confirm the existing `KC_SPI_X509CERT_LOOKUP_NGINX_*` env vars are in the Helm
|
||||
values (the new provider inherits them automatically — do NOT remove them).
|
||||
2. Hand the F5 team the one-pager (README §6) so their change request runs in parallel.
|
||||
Ask them for: cert SHA-256 fingerprint, subject DN, issuing CA chain (PEM), and the
|
||||
self-IP/SNAT CIDR the BigIP uses toward the Keycloak NLB.
|
||||
3. Confirm with networking whether the NLB target group preserves client IPs
|
||||
(needed only if you plan to use the CIDR trust rule; Phase 2 logs verify it either way).
|
||||
4. Brokerage-user readiness (should already be true): users exist/are federated in
|
||||
Keycloak, and their certs chain to a CA in the truststore.
|
||||
|
||||
## Phase 1 — Build the jar and image
|
||||
|
||||
```bash
|
||||
cd keycloak-hybrid-x509-spi
|
||||
mvn package # or the podman one-liner in README §1
|
||||
ls target/keycloak-hybrid-x509-spi-1.0.0.jar
|
||||
```
|
||||
|
||||
Add to the Keycloak image (see `Dockerfile.example` for the full picture):
|
||||
|
||||
```dockerfile
|
||||
COPY keycloak-hybrid-x509-spi-1.0.0.jar /opt/keycloak/providers/
|
||||
ENV KC_HTTPS_CLIENT_AUTH=request # BUILD-time option: must precede `kc.sh build`
|
||||
RUN /opt/keycloak/bin/kc.sh build
|
||||
```
|
||||
|
||||
> `request` = certificate **optional**. Never use `required` — it would break every
|
||||
> certless connection (OIDC backchannels, admin console, health checks).
|
||||
|
||||
## Phase 2 — Deploy in log mode *(zero expected behavior change)*
|
||||
|
||||
Helm values — add **only** these (leave every existing `..._NGINX_...` var untouched):
|
||||
|
||||
```yaml
|
||||
extraEnv: # or however your chart names it
|
||||
- name: KC_SPI_X509CERT_LOOKUP_PROVIDER
|
||||
value: hybrid
|
||||
- name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
|
||||
value: log
|
||||
- name: KC_LOG_LEVEL
|
||||
value: INFO,com.example.keycloak.x509:debug
|
||||
```
|
||||
|
||||
Deploy, then verify **in this order**:
|
||||
|
||||
1. Startup config was parsed correctly:
|
||||
```bash
|
||||
kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid: initialized"
|
||||
kubectl logs -n <ns> <keycloak-pod> | grep "inherited from the nginx provider config"
|
||||
```
|
||||
The `header=` value must match what the F5 injects. A WARN about "no trusted proxy
|
||||
rules" is expected at this phase.
|
||||
2. [ ] **Regression check:** a normal user logs in through the F5 with their CAC. Must
|
||||
work identically to before.
|
||||
3. Harvest values for later phases from the login's log lines:
|
||||
```bash
|
||||
kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid:"
|
||||
```
|
||||
- `remoteAddr=<ip>` → the F5's egress IP (→ `TRUSTED_PROXY_CIDRS` value; also tells
|
||||
you whether client-IP preservation is on — if you see NLB private IPs here, the
|
||||
CIDR rule is unusable and you must rely on the cert fingerprint).
|
||||
- `peer=no-tls-client-cert` → expected until Phase 3.
|
||||
- `header cert decoded using strategy '...'` (DEBUG) → confirms the F5's encoding.
|
||||
4. [ ] If a TGW-side tester is available: browse to
|
||||
`https://<keycloak-host>/realms/<realm>/account`, expect the browser cert-picker, pick
|
||||
the CAC cert, expect login. Log shows `using DIRECT TLS peer cert`.
|
||||
|
||||
**Rollback:** set `KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx`, redeploy. (Keep the image;
|
||||
`https-client-auth=request` is harmless on its own.)
|
||||
|
||||
## Phase 3 — F5 presents its client certificate
|
||||
|
||||
After the F5 team applies their Server SSL profile change:
|
||||
|
||||
1. [ ] Ensure the CA that issued the F5's client cert is in Keycloak's truststore
|
||||
(append to the bundle in `KC_TRUSTSTORE_PATHS` if it's a different CA). In `request`
|
||||
mode a client cert that fails validation aborts the handshake — coordinate cert +
|
||||
truststore in the same window if the CA is new.
|
||||
2. [ ] Regression check: F5 CAC login still works.
|
||||
3. Harvest the fingerprint (compare with what the F5 team reported):
|
||||
```bash
|
||||
kubectl logs -n <ns> <keycloak-pod> | grep -o 'peer=subject=\[[^]]*\] sha256=[a-f0-9]*' | sort -u
|
||||
```
|
||||
|
||||
**Rollback:** F5 team removes the client cert from the profile; everything else keeps working.
|
||||
|
||||
## Phase 4 — Enforce
|
||||
|
||||
Add the trust rules and flip the mode:
|
||||
|
||||
```yaml
|
||||
- name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CERT_SHA256
|
||||
value: "<sha256-from-phase-3>" # comma-separate old,new during F5 cert rotations
|
||||
- name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CIDRS
|
||||
value: "<f5-egress-cidr>" # optional second factor; omit if IPs weren't preserved
|
||||
- name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
|
||||
value: enforce
|
||||
```
|
||||
|
||||
Verify all three, in order:
|
||||
|
||||
1. [ ] F5 path: CAC login through the F5 works; log shows
|
||||
`using HEADER cert from trusted proxy (cert-fingerprint)`.
|
||||
2. [ ] Direct path: TGW-side CAC login works; log shows `using DIRECT TLS peer cert`.
|
||||
3. [ ] **Negative test (the point of this whole exercise):** from a TGW-side box, send a
|
||||
forged header and confirm it is rejected:
|
||||
```bash
|
||||
curl -vk "https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth?client_id=account-console&response_type=code&redirect_uri=https://<keycloak-host>/realms/<realm>/account/" \
|
||||
-H "<header-name>: $(cat any-user-cert.pem | python3 -c 'import sys,urllib.parse;print(urllib.parse.quote(sys.stdin.read()))')"
|
||||
```
|
||||
Expected: NO authenticated session, and the log shows
|
||||
`IGNORED cert header from untrusted peer`. If instead you see the login succeed,
|
||||
**stop** — the trust rules are wrong; re-check the fingerprint and revert to `log`.
|
||||
|
||||
**Rollback:** `KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=log` (keeps both paths working
|
||||
while you investigate), or provider back to `nginx` (F5 path only, pre-project behavior).
|
||||
|
||||
## Phase 5 — Quiet down (housekeeping)
|
||||
|
||||
```yaml
|
||||
- name: KC_SPI_X509CERT_LOOKUP_HYBRID_VERBOSE
|
||||
value: "false"
|
||||
- name: KC_LOG_LEVEL
|
||||
value: INFO
|
||||
```
|
||||
|
||||
Also confirm (once, not per-deploy):
|
||||
- [ ] OCSP/CRL revocation checking is enabled in the realm's X509 authenticator config
|
||||
(Authentication → browser x509 flow → config). The F5 pre-screens users on its path;
|
||||
the direct path relies entirely on this setting.
|
||||
- [ ] A calendar/reminder exists for the F5 client-cert expiry: add the *next* cert's
|
||||
fingerprint to `TRUSTED_PROXY_CERT_SHA256` (comma-separated) **before** rotation.
|
||||
|
||||
---
|
||||
|
||||
## If something breaks
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---|---|---|
|
||||
| F5 logins fail right after Phase 2 deploy | header name mismatch (inheritance picked wrong value) | check the `initialized`/`inherited` startup lines; set `KC_SPI_X509CERT_LOOKUP_HYBRID_SSL_CLIENT_CERT` explicitly |
|
||||
| F5 logins fail right after Phase 3 | F5 client cert fails TLS validation (CA missing from truststore) | add issuing CA to the bundle, restart; or F5 team pulls the cert |
|
||||
| F5 logins fail right after Phase 4 | fingerprint/CIDR wrong or F5 rotated its cert | revert `TRUST_MODE=log`, re-harvest fingerprint from logs |
|
||||
| TGW users get no cert prompt | truststore CAs don't cover their issuing CA (browser filters the picker), or TLS isn't reaching Keycloak at L4 | verify CA bundle; verify NLB listener is TCP passthrough, not TLS |
|
||||
| TGW users get a prompt but login fails | user mapping — cert attribute doesn't resolve to a Keycloak user | check the X509 authenticator's user-identity mapping config |
|
||||
| Everything on fire | — | `KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx` + redeploy = exact pre-project behavior |
|
||||
|
||||
Full log-line reference: README §7. Every decision the provider makes is logged with the
|
||||
`x509-hybrid:` prefix while `verbose=true`.
|
||||
Reference in New Issue
Block a user