How-To · Kubernetes API Gateway

01Recipes

Twenty-five tested copy-paste patterns, grouped by intent. Click a card to expand. Filter by category, or hit Expand all to scan everything at once.

Routing & traffic 7 recipes

1.1 · New HTTPRoute on an existing hostname3 fields, longest-prefix match, mind the sectionName.

HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: my-app, namespace: gateway-demo }
spec:
  parentRefs:
    - name: gateway-demo
      namespace: gateway-demo
      sectionName: https      # pin to one listener!
  hostnames: [ "gateway.wolfslight.cc" ]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /my-app }
      backendRefs:
        - name: my-app
          port: 80

kubectl -n gateway-demo get httproute my-app -o jsonpath='{.status.parents[0].conditions[?(@.type=="Accepted")].status}' → True

Without sectionName the route attaches to EVERY compatible listener — including http :80 — and races the HTTP→HTTPS redirect by longest-prefix-match.

There is no Gateway named eg, on any namespace. eg is this cluster's GatewayClass — every app has its own Gateway in its own namespace (kubectl get gateways -A). Point parentRefs at the app's real Gateway/namespace, not at the class name.

A wrong parentRef doesn't fail loud. If the referenced Gateway/namespace/sectionName doesn't exist, you will not see Accepted=False — the Gateway API spec doesn't define a status entry for an unresolvable parent at all, so status.parents just stays []. Nothing to grep for; check that the array has an entry before you check its condition.

Cross-namespace also needs allowedRoutes. 7 of this cluster's 8 Gateways have every listener set to allowedRoutes.namespaces.from: Same (the exception is envoy-gateway-system/platform, which uses Selector) — a Route sitting in a different namespace than the Gateway won't attach at all until that's changed to Selector or All.

1.2 · Canary / weighted traffic splitTwo backends behind one rule. Weights are integers, summed any number.

HTTPRoutecanary

  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - { name: api-v1, port: 80, weight: 90 }
        - { name: api-v2, port: 80, weight: 10 }    # 10% canary

for i in $(seq 1 100); do curl -s https://gateway.wolfslight.cc/api/ -H "X-Hello: 1" | jq -r .pod; done | sort | uniq -c

Weights are per-request, not sticky — a single user can hop versions. Combine with header-match rule + x-canary: true for sticky canary.

1.3 · URLRewrite — strip a path prefixFrontend says /api/users/42, backend sees /users/42.

HTTPRoutefilter

  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /            # strip /api
      backendRefs:
        - { name: api-svc, port: 80 }

curl https://gateway.wolfslight.cc/api/users/42 → backend's /users/42 handler fires.

ReplaceFullPath rewrites to a fixed path (no captures). For Envoy's regex-rewrite use EnvoyExtensionPolicy with a Lua filter.

1.4 · Request redirect (301 / 302)Move a path, force HTTPS, or send a deprecated endpoint to its new home.

HTTPRoutefilter

  rules:
    - matches:
        - path: { type: PathPrefix, value: /old }
      filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            hostname: new.wolfslight.cc
            path: { type: ReplacePrefixMatch, replacePrefixMatch: /v2 }
            statusCode: 301

curl -I https://gateway.wolfslight.cc/old/x → 301 with Location: https://new.wolfslight.cc/v2/x

A redirect rule must NOT also have backendRefs — they're mutually exclusive per Gateway-API spec.

1.5 · Request mirror (fire-and-forget shadow traffic)Send a copy of every request to a staging backend without affecting the response.

HTTPRoutefilter

  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      filters:
        - type: RequestMirror
          requestMirror:
            backendRef: { name: api-staging, port: 80 }
      backendRefs:
        - { name: api-prod, port: 80 }

Mirror is fire-and-forget — the mirror's response is discarded. Tail logs/metrics in the mirror target to confirm it sees traffic.

1.6 · Cross-namespace backend (ReferenceGrant)Your Route in gateway-demo points to a Service in another namespace. Spec says no without a grant.

ReferenceGrant

# Lives in the TARGET namespace (the backend's namespace) — "backend-ns" below
# is a placeholder, like "my-app" elsewhere on this page: gateway-demo's own
# backends today all live in-namespace, so there is no live pair to copy verbatim.
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata: { name: allow-from-gateway-demo, namespace: backend-ns }
spec:
  from:
    - group: gateway.networking.k8s.io
      kind:  HTTPRoute
      namespace: gateway-demo
  to:
    - group: ""
      kind:  Service

kubectl get referencegrant -A confirms the CRD and kind are real and installed here (v1beta1) — one is live today at ocis-app/platform-gateway-tls, though for a Gateway→Secret pair, not this HTTPRoute→Service one. After applying, the HTTPRoute that previously had ResolvedRefs=False, RefNotPermitted flips to True within seconds.

This is separate from allowedRoutes (see 1.1/1.7): allowedRoutes decides whether your Route may attach to the Gateway itself from another namespace; ReferenceGrant decides whether it may reference a Service (or Secret) in another namespace. You can need either, both, or neither depending on where the Gateway, the Route, and the backend each live.

1.7 · Attach an HTTPRoute to a Gateway you don't ownFind the real target, read its allowedRoutes, only then write the Route.

HTTPRoutecross-namespace

1Find the actual Gateway and namespace — never guess eg/default, every app has its own:

kubectl get gateways -A
# NAMESPACE              NAME             CLASS   PROGRAMMED
# airgate-app            gateclear        eg      True
# coder                  coder            eg      True
# envoy-gateway-system   platform         eg      True
# gameforge-palworld     palworld         eg      True
# gameforge-platform     gameforge-auth   eg      True
# gateway-demo           gateway-demo     eg      True
# iagd-app               insel            eg      True
# secpol-demo            secpol-demo      eg      True

2Read what its listener actually allows, per listener, before writing sectionName or attempting a cross-namespace attach:

kubectl -n <ns> get gateway <name> \
  -o jsonpath='{range .spec.listeners[*]}{.name}{" from="}{.allowedRoutes.namespaces.from}{"\n"}{end}'
# gateway-demo/gateway-demo → http from=Same / https from=Same
# envoy-gateway-system/platform → http from=Selector / https from=Selector / https-tools from=Selector

3Match the value you got:

Same:     # your Route must live in the SAME namespace as the Gateway — no exceptions
Selector: # your Route's namespace needs the label the Gateway's namespaceSelector matches
All:      # any namespace may attach

4HTTPRoute, pinned to one listener:

spec:
  parentRefs:
    - name: gateway-demo
      namespace: gateway-demo
      sectionName: https

kubectl -n gateway-demo get httproute features -o jsonpath='{.status.parents[*].conditions[*].type}{"="}{.status.parents[*].conditions[*].status}{"\n"}'

→ Accepted ResolvedRefs=True True (this is the live features Route on this cluster).

7 of this cluster's 8 Gateways are Same — cross-namespace attachment is closed by default almost everywhere here. Only envoy-gateway-system/platform uses Selector. If you need to attach from a foreign namespace and the target is Same, the only fix is on the Gateway's side (switch to Selector/All) — you cannot override it from the Route.

allowedRoutes only gates the attachment. If backendRefs also crosses a namespace boundary (Route in A, Service in B), you additionally need a ReferenceGrant in B — see 1.6. The two checks are independent; passing one says nothing about the other.

TLS, DNS & hostnames 2 recipes

2.1 · Add a brand-new hostname (TLS + DNS + listener)Four moving parts; external-dns + cert-manager do most of it automatically.

Gatewaycert-managerexternal-dns

1Add a Listener to your app's own Gateway — e.g. gateway-demo/gateway-demo, edited at its GitOps source apps/gateway-demo/gateway.yaml (Flux Kustomization flux-system/gateway-demo applies it; there's no shared Gateway eg and no manifests/20-gateway.yaml — eg is only the GatewayClass):

- name: https-my-app
  protocol: HTTPS
  port: 443
  hostname: my-app.wolfslight.cc
  tls:
    mode: Terminate
    certificateRefs:
      - name: my-app-wolfslight-tls
  allowedRoutes: { namespaces: { from: Same } }

2Add a per-hostname Certificate, in the same namespace as the Gateway — every Certificate on this cluster lives next to its Gateway (e.g. gateway-demo/gateway-wolfslight-tls), never in default:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: my-app-wolfslight-tls, namespace: gateway-demo }
spec:
  secretName: my-app-wolfslight-tls
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
  dnsNames: [ my-app.wolfslight.cc ]

3HTTPRoute referencing parentRefs: name: gateway-demo, namespace: gateway-demo, sectionName: https-my-app + the same hostname.

4DNS is automatic — external-dns syncs from Gateway/HTTPRoute hostnames to Cloudflare within ~30s.

kubectl -n gateway-demo get certificate my-app-wolfslight-tls → READY=True (DNS-01 takes 3-5 min)

Per-hostname certs avoid the wildcard TXT-conflict trap: cert-manager + external-dns can both create _acme-challenge records, but only safely on distinct names.

2.2 · Enforce TLS 1.2+ and AEAD ciphersClientTrafficPolicy attaches to the Gateway, applies to every HTTPS listener.

ClientTrafficPolicy

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata: { name: tls-min-1-2, namespace: gateway-demo }
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind:  Gateway
      name:  gateway-demo
  tls:
    minVersion: "1.2"
    ciphers:
      - ECDHE-ECDSA-AES256-GCM-SHA384
      - ECDHE-RSA-AES256-GCM-SHA384
      - ECDHE-ECDSA-CHACHA20-POLY1305
      - ECDHE-RSA-CHACHA20-POLY1305

nmap --script ssl-enum-ciphers -p 443 gateway.wolfslight.cc → only TLSv1.2 + TLSv1.3 visible, all AEAD. A live equivalent already runs on this cluster (iagd-app/insel-client, targetRefs.name: insel) — same shape, different app, proof this pattern is more than theoretical here.

There is no Gateway eg in namespace default — targetRefs.name must be a real Gateway's name (kubectl get gateways -A), and the ClientTrafficPolicy lives in that Gateway's own namespace, same as every other tenant policy on this cluster.

Security 7 recipes

3.1 · CORS allowlistSame CRD for HTTPRoute and GRPCRoute — kind-agnostic.

SecurityPolicy

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata: { name: cors-my-app, namespace: demo }
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind:  HTTPRoute           # or GRPCRoute
      name:  my-app
  cors:
    allowOrigins:  [ "https://gateway.wolfslight.cc" ]
    allowMethods:  [ GET, POST, OPTIONS ]
    allowHeaders:  [ "Content-Type", "Authorization" ]
    exposeHeaders: [ "x-request-id" ]
    maxAge: "1h"

curl -i -X OPTIONS https://my-app.wolfslight.cc/ -H 'Origin: https://gateway.wolfslight.cc' -H 'Access-Control-Request-Method: POST'

→ 200 + access-control-allow-origin header.

3.2 · Basic Auth on a pathSecret with htpasswd file; SecurityPolicy points to it.

SecurityPolicyhtpasswd

# Create secret
htpasswd -nbB admin "PleaseChangeMe" | \
  kubectl -n demo create secret generic my-app-basic-auth \
    --from-file=.htpasswd=/dev/stdin --dry-run=client -o yaml | \
  kubectl apply -f -

# Attach policy
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: my-app }
  basicAuth:
    users:
      name: my-app-basic-auth           # Secret name

curl -i https://my-app.wolfslight.cc/ → 401 · curl -u admin:PleaseChangeMe ... → 200.

3.3 · JWT verification (OIDC providers, Auth0, Keycloak)Envoy fetches the JWKS, verifies the signature, exposes claims as headers.

SecurityPolicyJWT

spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: my-api }
  jwt:
    providers:
      - name: auth0
        issuer: "https://my-tenant.auth0.com/"
        audiences: [ "https://api.wolfslight.cc" ]
        remoteJWKS:
          uri: "https://my-tenant.auth0.com/.well-known/jwks.json"
        claimToHeaders:
          - { claim: sub, header: x-user-id }
          - { claim: email, header: x-user-email }

curl -i https://my-api.wolfslight.cc/private → 401 · with Authorization: Bearer <valid JWT> → 200, backend sees x-user-id header.

Envoy caches the JWKS — key rotation propagates in ~5 min by default. jwks.cacheDuration overrides.

3.4 · IP allowlist / denylistSecurityPolicy with authorization.rules. CIDR-based.

SecurityPolicy

spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: admin-panel }
  authorization:
    defaultAction: Deny           # everything else → 403
    rules:
      - name: office-cidr
        action: Allow
        principal:
          clientCIDRs: [ "203.0.113.0/24", "198.51.100.42/32" ]

Source IP is the real client IP only if the Envoy data-plane Service has externalTrafficPolicy: Local (EG default). With Cluster mode you see the node's IP — useless for filtering.

3.5 · Inject security headers (HSTS, CSP, X-Frame-Options)ResponseHeaderModifier filter on the HTTPRoute — no policy needed.

HTTPRoutefilter

  rules:
    - filters:
        - type: ResponseHeaderModifier
          responseHeaderModifier:
            add:
              - { name: Strict-Transport-Security, value: "max-age=31536000; includeSubDomains; preload" }
              - { name: X-Frame-Options,           value: DENY }
              - { name: X-Content-Type-Options,    value: nosniff }
              - { name: Referrer-Policy,           value: "strict-origin-when-cross-origin" }
              - { name: Content-Security-Policy,   value: "default-src 'self'" }
      backendRefs:
        - { name: my-app, port: 80 }

Header values can't contain non-ASCII (RFC 7230). Don't put ·, →, em-dashes etc. — Envoy rejects with cryptic error.

3.6 · CCE ELB listener ACL — IP whitelist for L4 exposuresFor TCPRoute / UDPRoute backends where SecurityPolicy doesn't apply. Defense-in-depth layer 2.

CCE-CCMdefense-in-depth

1Find your ELB and listener IDs (the ELB id is on the Envoy data-plane Service annotations) — the label is your app's own Gateway name, never eg (that's only the GatewayClass):

kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=<gateway-name> \
  -o jsonpath='{.items[0].metadata.annotations.kubernetes\.io/elb\.id}'
# e.g. owning-gateway-name=palworld — this cluster's only non-HTTP(S) listener today (UDP, not TCP)

2Get your own current public IP (becomes the whitelist entry):

curl -s4 https://ifconfig.me    # → 193.0.113.42

3In OTC Console → Elastic Load Balance → your ELB → Listeners → the TCP listener (e.g. :6379) → Access Control:

Type:        Whitelist
IP group:    create new — name e.g. "office-only"
CIDR:        193.0.113.42/32
Apply to:    this listener only (NOT the whole ELB!)

From your laptop: nc -zv -w 5 <ELB-EIP> 6379 → succeeds · From a different network (mobile hotspot, VPN): same command → connection times out.

NEVER apply the ACL to the whole ELB — that would lock you out of :80 and :443 too. OTC ELB ACL is per-listener; insist on that scope.

Layer 2, not Layer 1. This ACL is the second line of defense. The first line is still backend self-auth (Redis requirepass, Postgres pg_hba.conf, etc.). If an attacker compromises an IP inside your whitelist, AUTH is what's left between them and your data.

3.7 · Verify a network ACL from outside (no second cloud account)Use a public probe service to test that your firewall rule actually drops non-whitelisted traffic.

verificationdefense-in-depth

Testing a firewall change from your own machine proves it doesn't lock you out — but it can't prove that anyone else is actually blocked. check-host.net runs probes from globally distributed nodes; if all of them time out while your laptop succeeds, your ACL is doing its job.

1Trigger a TCP probe against your restricted port from 5 external nodes:

REQ=$(curl -sS -H 'Accept: application/json' \
  "https://check-host.net/check-tcp?host=<EIP>:6379&max_nodes=5" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['request_id'])")
sleep 8                                          # let probes complete
curl -sS -H 'Accept: application/json' \
  "https://check-host.net/check-result/$REQ" | python3 -m json.tool

Every external node reports {"error": "Connection timed out"} while your own nc -zv -w 5 <EIP> 6379 still succeeds. → ACL is correctly scoped to allow you and block everyone else.

2Optional control test — same script against :443 (which should remain OPEN):

curl -sS "https://check-host.net/check-tcp?host=<EIP>:443&max_nodes=5" ...

All 5 external nodes report a time in seconds (e.g. 0.024) — :443 is publicly reachable. Confirms the ACL is per-listener, not ELB-wide.

3Real-user test — switch network (phone hotspot, VPN, friend's WiFi), then try the same nc -zv:

nc -zv -w 6 <EIP> 6379    # should TIMEOUT on the new IP
nc -zv -w 6 <EIP> 443     # should still SUCCEED

OTC ELB ACL blocks via silent packet drop, not TCP RST. That's why you see timeout not connection refused. Bonus: doesn't leak the existence of your filter rule to scanners — they can't distinguish "no service" from "ACL drop".

Don't use this against ports you don't own. check-host.net is for verification of your own infrastructure; abusing it as a recon tool against third-party endpoints is what gets the service rate-limited or shut down for everyone.

Reliability 3 recipes

4.1 · Per-IP rate limit (no Redis needed)BackendTrafficPolicy with type: Local — in-process per Envoy replica.

BackendTrafficPolicy

spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: my-app }
  rateLimit:
    type: Local
    local:
      rules:
        - clientSelectors:
            - sourceCIDR: { value: "0.0.0.0/0", type: Distinct }
          limit: { requests: 10, unit: Second }

for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code}\n" https://gateway.wolfslight.cc/my-app/; done | sort | uniq -c

→ 10× 200, 10× 429.

Local is per Envoy replica. 2 pods × 10rps = 20rps total budget. For cluster-wide use Global + Redis.

4.2 · Timeouts and retriesPer-route. Defaults are too generous for most APIs (15s + 1 retry = 30s worst-case).

BackendTrafficPolicy

spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: HTTPRoute, name: my-api }
  timeout:
    http:
      requestTimeout:          5s
      connectionIdleTimeout:   30s
  retry:
    numRetries: 2
    perRetry:
      timeout: 2s
      backOff: { baseInterval: 100ms, maxInterval: 1s }
    retryOn:
      triggers: [ 5xx, reset, connect-failure ]

Retries on POST are dangerous (non-idempotent). Either set numRetries: 0 or restrict to retriable-status-codes like 503-only.

4.3 · Circuit breaker (cap pending requests + connections)Saves a slow backend from being asphyxiated by retries.

BackendTrafficPolicy

spec:
  targetRefs:
    - { group: "", kind: Service, name: flaky-backend }
  circuitBreaker:
    maxConnections:      100
    maxPendingRequests:  50
    maxParallelRequests: 200
    maxParallelRetries:  5

Load-test to saturation; check Envoy stats circuit_breakers.default.cx_open ticks up — Envoy rejects with 503 LO instead of overloading backend.

Protocols 2 recipes

5.1 · gRPC service (HTTP/2 upstream)Service announces h2c via appProtocol, GRPCRoute attaches to an existing HTTPS listener.

GRPCRouteh2c

apiVersion: v1
kind: Service
metadata: { name: my-grpc, namespace: gateway-demo }
spec:
  selector: { app: my-grpc }
  ports:
    - name: grpc
      port: 9000
      appProtocol: kubernetes.io/h2c     # the magic — HTTP/2 upstream
---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata: { name: my-grpc, namespace: gateway-demo }
spec:
  parentRefs:
    - { name: gateway-demo, namespace: gateway-demo, sectionName: https }
  hostnames: [ "gateway.wolfslight.cc" ]
  rules:
    - backendRefs:
        - { name: my-grpc, port: 9000 }

grpcurl gateway.wolfslight.cc:443 list — your services appear.

There is no Gateway eg, no namespace default, and no dedicated https-grpc listener anywhere on this cluster — and you don't need one: every Gateway's https listener already lists GRPCRoute in supportedKinds (verified: kubectl -n gateway-demo get gateway gateway-demo -o jsonpath='{.status.listeners}'). Attach directly to the app's existing https listener with the Gateway's own hostname — a separate listener is only worth it if you want gRPC on its own hostname.

Without appProtocol: kubernetes.io/h2c Envoy falls back to HTTP/1.1 upstream → gRPC returns UNKNOWN. Browser calls? Use application/grpc-web-text — fetch can't read HTTP/2 trailers.

5.2 · Raw TCP service (Redis, Postgres, MQTT)Non-HTTP listener + TCPRoute. Port-based, no hostname.

TCPRouteexperimental

1Listener on your app's own Gateway — e.g. gateway-demo/gateway-demo, added in apps/gateway-demo/gateway.yaml. This is a real new addition: no Gateway on this cluster has a TCP listener today (the only non-HTTP(S) listeners in production are protocol: UDP, on gameforge-palworld/palworld):

- name: tcp-pg
  protocol: TCP
  port: 5432
  allowedRoutes:
    kinds: [ { kind: TCPRoute } ]
    namespaces: { from: Same }

2TCPRoute (experimental channel):

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata: { name: pg, namespace: gateway-demo }
spec:
  parentRefs:
    - { name: gateway-demo, namespace: gateway-demo, sectionName: tcp-pg }
  rules:
    - backendRefs:
        - { name: postgres, port: 5432 }

Scan your own backend with nmap from outside BEFORE the first bot finds it. TCPRoute is L4 — the Gateway has no idea what protocol your backend speaks, so SecurityPolicy (CORS, BasicAuth, JWT, IP-allowlist) does nothing here. What you get for free with HTTPRoute, the backend has to do itself: Redis needs requirepass, Postgres needs pg_hba.conf, MQTT needs username/password or mTLS. Mental model: HTTP = Gateway-secured, TCP = Backend-secured.

No hostnames — port-based only. The ELB needs the external port open via the Envoy data-plane Service. There's no eg Gateway, no default namespace, and no standalone install script on this cluster — the experimental Gateway API channel (tcproutes.gateway.networking.k8s.io/v1alpha2, confirmed installed) comes in via the Flux-managed envoy-gateway HelmRelease (gateway-helm chart, flux-system), not a scripts/*.sh step. This is the one recipe on this page with no live counterpart to check against — test it in a scratch namespace before touching a real app's gateway.yaml.

Infrastructure & ELB 4 recipes

6.1 · Provision a standalone ELB (outside Gateway API)Plain Service: LoadBalancer with CCE annotations. New ELB or attach to existing one.

CCE-CCML4

# A) Autocreate new ELB + EIP
metadata:
  annotations:
    kubernetes.io/elb.class: union
    kubernetes.io/elb.autocreate: |
      {
        "type": "public",
        "name": "k8s-my-lb",
        "bandwidth_name": "k8s-my-lb-bw",
        "bandwidth_chargemode": "traffic",
        "bandwidth_size": 5,
        "bandwidth_sharetype": "PER",
        "eip_type": "5_bgp"
      }
spec:
  type: LoadBalancer

# B) Reuse an existing ELB (saves cost + EIP)
metadata:
  annotations:
    kubernetes.io/elb.class: union
    kubernetes.io/elb.id: "<existing-elb-id>"

CCM rejects autocreate JSON without bandwidth_name — webhook error is cryptic. Same field is mandatory for the Envoy Gateway data-plane Service.

6.2 · Inject custom request headers (Trace IDs, internal markers)RequestHeaderModifier filter — set/add/remove headers Envoy passes upstream.

HTTPRoutefilter

  rules:
    - filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - { name: x-injected-by-gateway, value: "true" }
              - { name: x-trace-id, value: "%REQ(x-request-id)%" }
            set:
              - { name: user-agent, value: "gateway-injected" }
            remove: [ "cookie" ]            # strip cookies upstream
      backendRefs:
        - { name: my-app, port: 80 }

Envoy interpolates %REQ(...)% / %DOWNSTREAM_REMOTE_ADDRESS% / etc. when EG translates the filter to its config. Plain Gateway-API doesn't define this, but EG accepts it.

6.3 · Attach a Gateway to an existing, externally-created ELBSame-VPC check first, then elb.id — and the one event that tells success from failure.

CCE-CCMelb.id

1Before anything else: is the ELB in the same VPC as the cluster's node subnet? This is the one check that decides whether the rest of this recipe works at all. If it doesn't match, the CCM will still report success on the surface (see step 3) while the ELB never gets a working member.

2Point the Envoy data-plane Service at the existing ELB (mutually exclusive with elb.autocreate):

metadata:
  annotations:
    kubernetes.io/elb.class: union
    kubernetes.io/elb.id: "<existing-elb-uuid>"

3Don't trust EnsuredLoadBalancer or a populated EXTERNAL-IP — both look identical whether the bind worked or not. Read the events for the actual member-creation result:

kubectl -n envoy-gateway-system describe svc <envoy-svc> | sed -n '/^Events:/,$p'
# success: Normal  UpdatedLoadBalancer          Updated load balancer with new pods
# failure: Warning UpdateLoadBalancerFailed    Failed to create member : {"NeutronError":
#          {"message": "Vpc <cluster-vpc> of member's subnet_cidr <subnet> and vpc
#          <elb-vpc> of loadbalancer <elb-id> mismatch", "detail": "", "type": "BadRequest"}},
#          status code: 400

The measured VPC-mismatch failure, verbatim: a 400 BadRequest from Neutron with exactly the message above, repeated on every reconcile — not a transient race. EXTERNAL-IP gets set regardless, to a private VIP nobody can reach.

Even when it works, EXTERNAL-IP only ever shows the private VIP on this path — never the ELB's public EIP, even with a bound EIP and even after manually setting kubernetes.io/elb.eip-id (measured to change nothing). external-dns reads exactly that field for its A-record; on this cluster its --exclude-target-net config makes it skip the private VIP and create no record at all, rather than write a wrong one — full writeup, the verified ranges, and the fix-in-progress at /loadbalancer.

Two Gateways can share one ELB — differentiated purely by listener port, same public address. This is the actual reason to reach for elb.id instead of autocreate: two Gateways on ports 80 and 9090 against the same ELB, both reachable under the same EIP, no port conflict.

Teardown: deleting the Gateway/namespace does not delete a hand-created ELB — the CCM only manages the member/listener it added, not the ELB itself. The Envoy Service in envoy-gateway-system can linger for a few seconds after the owning Gateway is gone (asynchronous GC on the generated Deployment/Service) — that's expected teardown lag, not a leaked resource.

6.4 · A second Gateway with its own auto-created ELBThe pattern all eight Gateways on this cluster actually use.

CCE-CCMelb.autocreate

1Set on the EnvoyProxy CR's provider.kubernetes.envoyService.annotations (this is the literal block from apps/gateway-demo/gateway.yaml; apps/secpol-demo/gateway.yaml uses the identical shape):

envoyService:
  type: LoadBalancer
  externalTrafficPolicy: Local
  annotations:
    kubernetes.io/elb.class: union
    kubernetes.io/elb.autocreate: |
      {
        "type": "public",
        "name": "my-app-elb",
        "bandwidth_name": "my-app-bw",
        "bandwidth_chargemode": "traffic",
        "bandwidth_size": 5,
        "bandwidth_sharetype": "PER",
        "eip_type": "5_bgp"
      }

kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=gateway-demo -o wide → EXTERNAL-IP carries two comma-separated addresses (public EIP and private VIP — order isn't consistent across Gateways, so match on the address, not the position) — confirmed live on all eight Gateways on this cluster today. Also confirmed: the CCM writes kubernetes.io/elb.id back onto the Service after autocreate, so you can read the ELB's UUID off the running Service without going to the OTC console.

bandwidth_name is mandatory — the CCM (and, separately, the admission webhook on the Envoy data-plane Service) rejects the annotation JSON without it, with a cryptic error either way.

This is the opposite trade-off from 6.3: a dedicated ELB per app (no port-sharing to reason about, no VPC pre-check needed since the CCM provisions the ELB inside the cluster's own VPC) at the cost of one ELB + one EIP per Gateway. All eight Gateways on this cluster take that trade.

02kubectl cheatsheet

The exact one-liners we use when something doesn't behave. Group by intent, not by resource — most debugging starts with "what state is this Route in?".

Intent	Command
Switch to the right context (insecure-skip-tls-verify is NEVER what you want)	kubectl config use-context wolf-test-k8s-api-gw-v2-external
What Routes exist, any kind, any namespace	kubectl get httproute,grpcroute,tcproute -A -o wide
Route accepted? backends resolved?	kubectl -n gateway-demo get httproute features -o jsonpath='{.status.parents[].conditions[].type}{"="}{.status.parents[].conditions[].status}{"\n"}'
Live Gateway status (Programmed=True is the gate)	kubectl get gateway -A -o wide
What EG-extension policies attach to a Route?	kubectl get securitypolicy,backendtrafficpolicy,clienttrafficpolicy,envoyextensionpolicy -A
Envoy data-plane logs (where Coraza WAF / rate-limit denials show up)	kubectl -n envoy-gateway-system logs -l app.kubernetes.io/name=envoy --tail=200 -f
Envoy Gateway controller logs (CRD reconcile errors)	kubectl -n envoy-gateway-system logs deploy/envoy-gateway --tail=200 -f
*Restart your app's* Envoy data plane after listener changes** — no selector restarts every app's data plane and the controller in one shot	kubectl -n envoy-gateway-system rollout restart deploy -l gateway.envoyproxy.io/owning-gateway-name=gateway-demo
Show all Certificates and whether they're ready	kubectl get certificate -A
Force a cert reissue (debug Let's Encrypt issues)	kubectl -n gateway-demo annotate certificate <name> cert-manager.io/issue-temporary-certificate-
external-dns: what does it see + what's it doing?	kubectl -n external-dns logs -l app.kubernetes.io/name=external-dns --tail=100
The ELB IP of the Gateway (without scraping the UI)	kubectl get svc -A -l gateway.envoyproxy.io/owning-gateway-name=gateway-demo -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}'
Which pod backs my Service right now?	kubectl -n gateway-demo get endpoints my-svc -o yaml
Hit a Service directly (skip the ELB, sanity-check the pod)	kubectl -n demo run curl --rm -i --image=curlimages/curl -- curl -sv http://my-svc/
What's on the cluster — full inventory	kubectl get gateway,httproute,grpcroute,tcproute,securitypolicy,backendtrafficpolicy,clienttrafficpolicy,envoyextensionpolicy -A
*Apply UI changes after editing `site/.html`** — no build script here, Flux applies the ConfigMap on its own schedule	flux reconcile kustomization gateway-demo --with-source -n flux-system

03Debug decision tree

Three of the most common failure modes, in the order we actually check them. Each step is a question — answer it before moving on.

"My Route doesn't work — empty reply / 404 / wrong backend"

Is the Gateway Programmed=True?
kubectl -n gateway-demo get gateway gateway-demo -o yaml | yq '.status.conditions' — there is no Gateway named eg in any namespace (eg is the GatewayClass); use your app's own Gateway/namespace, found via kubectl get gateways -A. If false, the EnvoyProxy CR or listener config is rejected — check controller logs.
Is the Route Accepted=True AND ResolvedRefs=True?
Accepted=False usually means a hostname conflict or missing sectionName. ResolvedRefs=False means a backend Service or cross-namespace ReferenceGrant is missing.
Does the path match what you think it matches?
Envoy uses longest-prefix-match on path. /foo/bar wins over /foo. Header/method matchers are ANDed within a rule.
Does the backend Service have endpoints?
kubectl get endpoints <svc> — if empty, the Service selector doesn't match any pod labels.
Still broken? Curl directly:
(a) from inside the cluster kubectl run curl --rm -i --image=curlimages/curl -- curl -sv http://svc.namespace.svc.cluster.local/ (skips Envoy) — if this fails, it's a pod/service problem, not Gateway. (b) from your laptop with --resolve to bypass DNS.

"TLS handshake fails / browser shows cert warning"

Is the Certificate Ready=True?
kubectl get certificate -A. If pending, Let's Encrypt is either rate-limited (use staging) or DNS-01 challenge is failing.
Does the Listener reference the right Secret name?
Cert-manager creates a Secret with the name from spec.secretName. Listener's tls.certificateRefs[0].name must match.
Is the cert in the SAME namespace as the Gateway?
Default behavior: certificateRefs only resolve in-namespace. Cross-namespace requires a ReferenceGrant.
Cert looks right, browser still warns?
Check openssl s_client -servername <hostname> -connect <hostname>:443 — issuer should be R10/R11/R12/R13 (Let's Encrypt). If you see (STAGING), your ClusterIssuer is pointing at the staging URL.

"Browser fetch fails with CORS / Load failed"

Is there a SecurityPolicy.cors attached to the Route?
kubectl get securitypolicy -A | grep cors. Without it, every cross-origin POST with a non-simple Content-Type triggers a preflight that gets 404.
Does allowOrigins include the calling origin EXACTLY?
Origins are matched verbatim — https://example.com doesn't match https://example.com:443. Wildcards via "*" only work without credentials.
Does allowHeaders include every custom header you send?
The browser preflight lists the actual headers — Envoy will reject if any is missing from the policy.
Browser still cached the old preflight?
maxAge: 1h means the browser keeps the old (failed) preflight for up to an hour. Disable cache in DevTools, or open a private tab.

04Glossary

The terms that trip people up — especially folks coming from Ingress.

parentRef

The thing a Route attaches to. Almost always a Gateway. Pinning sectionName further targets one specific listener on that Gateway.

sectionName

Names a specific listener on the Gateway. Without it, a Route attaches to every compatible listener — including http :80, which often isn't what you want.

allowedRoutes

Per-listener gate: which namespaces and Route kinds may attach? Use kinds: [GRPCRoute] to reserve a listener for gRPC, etc. On this cluster, 7 of 8 Gateways set namespaces.from: Same — cross-namespace attachment is closed by default almost everywhere here; only envoy-gateway-system/platform uses Selector.

appProtocol

Tells Envoy what protocol to speak upstream. kubernetes.io/h2c = HTTP/2 cleartext (essential for gRPC backends). Default is HTTP/1.1.

ReferenceGrant

Cross-namespace permission. A Route in namespace A pointing to a Service in namespace B needs a ReferenceGrant in B that permits it. Required by spec, not just policy.

EnvoyProxy CR

Envoy-Gateway-specific. Controls how the data plane is provisioned — replicas, resources, and the ELB annotations baked into the Service.

SecurityPolicy

CORS, BasicAuth, JWT, OIDC, ext-auth, API-key. Single CRD covers all of them. Attaches via targetRefs to HTTPRoute or GRPCRoute — kind-agnostic.

BackendTrafficPolicy

Per-backend behaviour: rate limit, retries, timeouts, fault injection, circuit breaking. Attaches to a Route or Service.

ClientTrafficPolicy

Per-listener client-side knobs: TLS minimum, allowed ciphers, HTTP/3 enablement, proxy-protocol, max-concurrent-streams.

EnvoyExtensionPolicy

Inject WASM or Lua filters into Envoy. Backs a Coraza WAF on this cluster — in secpol-demo, not gateway-demo. Use when no native CRD exists for what you need.

autocreate vs elb.id

CCE-specific. autocreate = a new ELB+EIP per Service (the pattern all eight production Gateways here use). elb.id = attach to an existing ELB, differentiated by listener port — the way to share one ELB across Gateways, see 6.3/6.4. They report differently, too: autocreate Services show both the public EIP and the private VIP in EXTERNAL-IP; a hand-attached elb.id Service shows only the private VIP, even with a bound EIP — on this cluster that makes external-dns skip creating a record for it entirely (see /loadbalancer).

grpc-web vs gRPC

Native gRPC uses HTTP/2 trailers — browsers can't read them via fetch(). grpc-web puts status inline in the body. Envoy Gateway translates between them transparently on the same GRPCRoute.

05Repo layout

This cluster is Flux-managed, not a Bash-script deploy — there's no scripts/*.sh install pipeline and no single Gateway with "6 listeners". Every real Gateway here (there are eight) has between 1 and 4 listeners, HTTP/HTTPS/UDP only — no TCP listener exists on this cluster today. Where things actually live, for this app:

clusters/wolf-apigw-external/gateway-demo.yaml	The Flux `Kustomization` (`flux-system/gateway-demo`) that reconciles this app — impersonated (`demo-reconciler` SA), deliberately not listed in the generic `./apps` Kustomization to avoid a prune conflict.
apps/gateway-demo/kustomization.yaml	Resource list + `configMapGenerator` that bundles every `site/*.html` page (including this one) into the `gateway-demo-site` ConfigMap nginx serves.
apps/gateway-demo/namespace.yaml	The `gateway-demo` namespace itself (Capsule tenant-owned).
apps/gateway-demo/gateway.yaml	EnvoyProxy CR (resources, `elb.autocreate` annotations — see 6.4) + the Gateway (`gatewayClassName: eg`, 2 listeners: `http`/`https`) + its cert-manager Certificate.
apps/gateway-demo/httproutes.yaml	The real HTTPRoutes: `echo`, `features`, `secret`, `gateway-demo-https-redirect` — all `parentRefs: name: gateway-demo, sectionName: https`.
apps/gateway-demo/policies.yaml	Two SecurityPolicies (`cors-features`, `secret-basic-auth`) + one BackendTrafficPolicy (`gateway-demo-ratelimit`) — the live counterparts recipes 3.1, 3.2 and 4.1 above cite.
apps/gateway-demo/basic-auth-secret.yaml	The htpasswd Secret backing the Basic-Auth SecurityPolicy.
apps/gateway-demo/networkpolicies.yaml	Cilium NetworkPolicies scoping the namespace.
apps/gateway-demo/deployments.yaml	Backends: echo, echo-v2, hello, slow, secret-page, ui.
apps/gateway-demo/configmap-*.yaml	Static content ConfigMaps for the hello and secret-page backends.
apps/gateway-demo/site-nginx.conf	nginx config serving the `site/*.html` ConfigMap.
apps/gateway-demo/site/*.html	All UI pages — this file included. Edit here; the Flux Kustomization applies the generated ConfigMap on the next reconcile (no manual "build" step).
infrastructure/controllers/envoy-gateway/release.yaml	The Flux `HelmRelease` that installs Envoy Gateway itself (`gateway-helm` chart, `v1.7.0`). Its Helm `releaseName` is also `eg` (whether that's the reason the GatewayClass got the same name wasn't checked here — just don't assume "`eg`" ever refers to a Gateway). Gateway API CRDs (standard + experimental, incl. `tcproutes`/`grpcroutes`) ship with this chart — there's no separate install script.
infrastructure/configs/gateway-class.yaml	The shared `GatewayClass eg` + the EnvoyProxy config backing the `envoy-gateway-system/platform` Gateway — the one Gateway on this cluster using `allowedRoutes.namespaces.from: Selector` instead of `Same`.