# Runbook: Rolling out the Hybrid X509 Lookup SPI

Step-by-step deployment procedure for operators. Background, config reference, and
troubleshooting tables live in [README.md](README.md) — this document is the "what do I
actually type" guide.

**What we're changing and why (30 seconds):** Today all user traffic passes through the
F5 BigIP, which validates user CAC/x509 certs and forwards the cert to Keycloak in an HTTP
header. New Transit Gateway (TGW) connectivity from brokerage AWS accounts bypasses the F5,
so Keycloak must additionally accept certs **directly from the TLS handshake**, while
making sure the F5's cert header **can't be forged** by TGW clients. This SPI does both.

```
Today:            user ──mTLS──> F5 (validates cert, adds header) ──TLS──> NLB(L4) ──> Keycloak
New TGW path:     user ────────────────────mTLS───────────────────────────> NLB(L4) ──> Keycloak
```

---

## Roles and coordination

| Who | Needed for | Task |
|---|---|---|
| Keycloak/EKS operator | every phase | image build, Helm values, log checks |
| F5/BigIP team | Phase 3 | add client cert to the Server SSL profile (see README §6 one-pager) |
| TGW-side tester | Phases 2 & 4 | a browser + CAC on a machine that reaches Keycloak *without* the F5 |

Each phase is designed to fit one ~30-minute test window and to be independently
verifiable and reversible. **Do not combine phases** — the log output of each phase is the
input to the next.

---

## Phase 0 — Prep (no deployment)

1. Collect from the current deployment:
   - [ ] Keycloak version → set `<keycloak.version>` in `pom.xml` to match (24.x–26.x).
   - [ ] Base image tag and whether the entrypoint uses `start --optimized`.
   - [ ] Truststore config: `KC_TRUSTSTORE_PATHS` (KC 25+) or `spi-truststore-file-*`
         env vars. The user-CA bundle must be present (it already is if header login works).
   - [ ] Confirm the existing `KC_SPI_X509CERT_LOOKUP_NGINX_*` env vars are in the Helm
         values (the new provider inherits them automatically — do NOT remove them).
2. Hand the F5 team the one-pager (README §6) so their change request runs in parallel.
   Ask them for: cert SHA-256 fingerprint, subject DN, issuing CA chain (PEM), and the
   self-IP/SNAT CIDR the BigIP uses toward the Keycloak NLB.
3. Confirm with networking whether the NLB target group preserves client IPs
   (needed only if you plan to use the CIDR trust rule; Phase 2 logs verify it either way).
4. Brokerage-user readiness (should already be true): users exist/are federated in
   Keycloak, and their certs chain to a CA in the truststore.

## Phase 1 — Build the jar and image

```bash
cd keycloak-hybrid-x509-spi
mvn package                        # or the podman one-liner in README §1
ls target/keycloak-hybrid-x509-spi-1.0.0.jar
```

Add to the Keycloak image (see `Dockerfile.example` for the full picture):

```dockerfile
COPY keycloak-hybrid-x509-spi-1.0.0.jar /opt/keycloak/providers/
ENV KC_HTTPS_CLIENT_AUTH=request   # BUILD-time option: must precede `kc.sh build`
RUN /opt/keycloak/bin/kc.sh build
```

> `request` = certificate **optional**. Never use `required` — it would break every
> certless connection (OIDC backchannels, admin console, health checks).

## Phase 2 — Deploy in log mode  *(zero expected behavior change)*

Helm values — add **only** these (leave every existing `..._NGINX_...` var untouched):

```yaml
extraEnv:                          # or however your chart names it
  - name: KC_SPI_X509CERT_LOOKUP_PROVIDER
    value: hybrid
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
    value: log
  - name: KC_LOG_LEVEL
    value: INFO,com.example.keycloak.x509:debug
```

Deploy, then verify **in this order**:

1. Startup config was parsed correctly:
   ```bash
   kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid: initialized"
   kubectl logs -n <ns> <keycloak-pod> | grep "inherited from the nginx provider config"
   ```
   The `header=` value must match what the F5 injects. A WARN about "no trusted proxy
   rules" is expected at this phase.
2. [ ] **Regression check:** a normal user logs in through the F5 with their CAC. Must
   work identically to before.
3. Harvest values for later phases from the login's log lines:
   ```bash
   kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid:"
   ```
   - `remoteAddr=<ip>` → the F5's egress IP (→ `TRUSTED_PROXY_CIDRS` value; also tells
     you whether client-IP preservation is on — if you see NLB private IPs here, the
     CIDR rule is unusable and you must rely on the cert fingerprint).
   - `peer=no-tls-client-cert` → expected until Phase 3.
   - `header cert decoded using strategy '...'` (DEBUG) → confirms the F5's encoding.
4. [ ] If a TGW-side tester is available: browse to
   `https://<keycloak-host>/realms/<realm>/account`, expect the browser cert-picker, pick
   the CAC cert, expect login. Log shows `using DIRECT TLS peer cert`.

**Rollback:** set `KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx`, redeploy. (Keep the image;
`https-client-auth=request` is harmless on its own.)

## Phase 3 — F5 presents its client certificate

After the F5 team applies their Server SSL profile change:

1. [ ] Ensure the CA that issued the F5's client cert is in Keycloak's truststore
   (append to the bundle in `KC_TRUSTSTORE_PATHS` if it's a different CA). In `request`
   mode a client cert that fails validation aborts the handshake — coordinate cert +
   truststore in the same window if the CA is new.
2. [ ] Regression check: F5 CAC login still works.
3. Harvest the fingerprint (compare with what the F5 team reported):
   ```bash
   kubectl logs -n <ns> <keycloak-pod> | grep -o 'peer=subject=\[[^]]*\] sha256=[a-f0-9]*' | sort -u
   ```

**Rollback:** F5 team removes the client cert from the profile; everything else keeps working.

## Phase 4 — Enforce

Add the trust rules and flip the mode:

```yaml
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CERT_SHA256
    value: "<sha256-from-phase-3>"          # comma-separate old,new during F5 cert rotations
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CIDRS
    value: "<f5-egress-cidr>"               # optional second factor; omit if IPs weren't preserved
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
    value: enforce
```

Verify all three, in order:

1. [ ] F5 path: CAC login through the F5 works; log shows
   `using HEADER cert from trusted proxy (cert-fingerprint)`.
2. [ ] Direct path: TGW-side CAC login works; log shows `using DIRECT TLS peer cert`.
3. [ ] **Negative test (the point of this whole exercise):** from a TGW-side box, send a
   forged header and confirm it is rejected:
   ```bash
   curl -vk "https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth?client_id=account-console&response_type=code&redirect_uri=https://<keycloak-host>/realms/<realm>/account/" \
        -H "<header-name>: $(cat any-user-cert.pem | python3 -c 'import sys,urllib.parse;print(urllib.parse.quote(sys.stdin.read()))')"
   ```
   Expected: NO authenticated session, and the log shows
   `IGNORED cert header from untrusted peer`. If instead you see the login succeed,
   **stop** — the trust rules are wrong; re-check the fingerprint and revert to `log`.

**Rollback:** `KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=log` (keeps both paths working
while you investigate), or provider back to `nginx` (F5 path only, pre-project behavior).

## Phase 5 — Quiet down (housekeeping)

```yaml
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_VERBOSE
    value: "false"
  - name: KC_LOG_LEVEL
    value: INFO
```

Also confirm (once, not per-deploy):
- [ ] OCSP/CRL revocation checking is enabled in the realm's X509 authenticator config
  (Authentication → browser x509 flow → config). The F5 pre-screens users on its path;
  the direct path relies entirely on this setting.
- [ ] A calendar/reminder exists for the F5 client-cert expiry: add the *next* cert's
  fingerprint to `TRUSTED_PROXY_CERT_SHA256` (comma-separated) **before** rotation.

---

## If something breaks

| Symptom | Likely cause | Action |
|---|---|---|
| F5 logins fail right after Phase 2 deploy | header name mismatch (inheritance picked wrong value) | check the `initialized`/`inherited` startup lines; set `KC_SPI_X509CERT_LOOKUP_HYBRID_SSL_CLIENT_CERT` explicitly |
| F5 logins fail right after Phase 3 | F5 client cert fails TLS validation (CA missing from truststore) | add issuing CA to the bundle, restart; or F5 team pulls the cert |
| F5 logins fail right after Phase 4 | fingerprint/CIDR wrong or F5 rotated its cert | revert `TRUST_MODE=log`, re-harvest fingerprint from logs |
| TGW users get no cert prompt | truststore CAs don't cover their issuing CA (browser filters the picker), or TLS isn't reaching Keycloak at L4 | verify CA bundle; verify NLB listener is TCP passthrough, not TLS |
| TGW users get a prompt but login fails | user mapping — cert attribute doesn't resolve to a Keycloak user | check the X509 authenticator's user-identity mapping config |
| Everything on fire | — | `KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx` + redeploy = exact pre-project behavior |

Full log-line reference: README §7. Every decision the provider makes is logged with the
`x509-hybrid:` prefix while `verbose=true`.