Service → ELB · Direct LoadBalancer

1. What happens when you `kubectl apply`

Seven steps. Four actors: you, the K8s api-server, the OTC cloud-controller (hws-cloudprovider), and the OTC ELB API itself.

youApply the manifest

kubectl apply -f lb-demo.yaml — sends a Service object with type: LoadBalancer and a kubernetes.io/elb.autocreate annotation to the API server.

api-serverStores the Service

etcd holds the new Service. .status.loadBalancer.ingress is empty — there's no public address yet.

ccmSees the Service event

The OTC cloud-controller (hws-cloudprovider) watches Services cluster-wide. It spots type: LoadBalancer and reads the elb.* annotations to figure out what to provision.

ccmotc-apiCalls the ELB and EIP APIs

One API call creates the ELB instance, another creates the EIP. The EIP is bound to the ELB. A listener on the Service's port is added, pointing to a backend pool of cluster nodes.

ccmWrites back IDs as annotations

The CCM updates the Service object with kubernetes.io/elb.id and kubernetes.io/elb.eip-id. These become the link between the K8s object and the OTC resources.

ccmSets `.status.loadBalancer.ingress`

The public EIP appears in .status.loadBalancer.ingress[0].ip. kubectl get svc now shows it under EXTERNAL-IP. Anything else in the cluster that watches Services (kube-proxy, the Gateway controller, etc.) reacts.

otc-fabricBGP advertises the EIP

A few seconds later the EIP becomes routable on the public internet. Traffic to the EIP hits the ELB → a healthy node → the pod via kube-proxy.

2. Two patterns: create a new ELB or reuse an existing one

The Swiss OTC CCM honours two distinct annotations for shaping the ELB. Pick one per Service. With autocreate you get a brand-new ELB+EIP per Service; with elb.id you attach another listener to an ELB that already exists. Pattern A is alive on this cluster right now (all eight production Gateways use it); Pattern B was verified in an isolated spike, not on a production Service — see the callouts below for exactly what that spike did and did not prove.

Check this first: ELB and node subnet must share one VPC

This is the very first thing to verify before pointing elb.id at an existing ELB — before annotations, before listeners, before anything else. If the ELB lives in a different VPC than the cluster's node subnet, the member-binding step of the reconcile fails with a Neutron 400 BadRequest, repeated on every retry:

Warning  UpdateLoadBalancerFailed  hws-cloudprovider  Details: Update member of listener/pool(...)
error: Failed to create member : {"NeutronError": {"message": "Vpc ceb35a98-27fd-4ed0-bd2c-2690a417c75a
of member's subnet_cidr 74b3caaf-65e1-4379-9b24-8d4aa9ce74ff and vpc efd930dc-c446-4af3-9577-e2c513cad348
of loadbalancer 040d02f2-d06e-46a5-ae88-4d93ffecb636 mismatch", "detail": "", "type": "BadRequest"}}, status code: 400

Measured on this cluster (2026-08-26, spike ELB 040d02f2-…): the error repeated five times identically over ~4:41 min — a stable structural defect, not a transient race. And it does not stop the Service from looking successful: EXTERNAL-IP still gets set (see the next box). A same-VPC ELB (spike ELB 5a4dbe7b-…) reconciled cleanly with no such error.

Pattern A · autocreate (new ELB) Demo: lb-demo / hello

apiVersion: v1
kind: Service
metadata:
  name: hello
  namespace: lb-demo
  annotations:
    kubernetes.io/elb.class: "union"
    kubernetes.io/elb.autocreate: |
      { "type":      "public",
        "name":      "k8s-lb-demo",
        "bandwidth_name":       "k8s-lb-demo-bw",
        "bandwidth_chargemode": "traffic",
        "bandwidth_size":       5,
        "bandwidth_sharetype":  "PER",
        "eip_type":             "5_bgp" }
spec:
  type: LoadBalancer
  selector: { app: hello }
  ports:
    - { port: 80, targetPort: http }
# Result: EIP 138.124.232.123, brand-new ELB k8s-lb-demo

CCM action: +create ELB k8s-lb-demo +create EIP, attach to ELB +add listener :80 +write back elb.id and elb.eip-id annotations on the Service.

Pattern B · elb.id reference (share existing ELB) Demo: whoami / fortune

apiVersion: v1
kind: Service
metadata:
  name: fortune
  namespace: whoami
  annotations:
    kubernetes.io/elb.class: "union"
    kubernetes.io/elb.id: "71f02bde-...d43271"
    # NO autocreate — we're pointing at an existing ELB.
    # The UUID comes from the whoami Service the CCM
    # wrote back when whoami used Pattern A.
spec:
  type: LoadBalancer
  selector: { app: fortune }
  ports:
    # Different port → no listener conflict
    - { port: 8080, targetPort: http }
# Result: EXTERNAL-IP shows the ELB's PRIVATE VIP, e.g. 172.16.0.19 —
# never the public EIP. The listener is added and the ELB is shared;
# the field kubectl prints just isn't the address you'd expect.

CCM action: ~look up ELB by ID (no create) ~EIP unchanged +add listener :8080 to the existing ELB ~backend pool gets a second member group !writes the private VIP, not the EIP, into .status.loadBalancer.ingress

The elb.id UUID is normally read from the Service that already created the ELB, e.g. kubectl get svc whoami -o jsonpath='{.metadata.annotations.kubernetes\.io/elb\.id}', then substituted into the manifest before kubectl apply. Hardcoding the UUID would couple the manifest to one specific cluster. There is no packaged script for this in the repo today — treat the command above as the recipe, not as a wrapper you can run as-is.

⚠ external-dns + `elb.id`: no DNS record at all, and the reason isn't in DNS

external-dns reads exactly one field for the A-record: status.loadBalancer.ingress[].ip. On the elb.id path that field holds the private VIP (measured: 172.16.0.19) — never the public EIP, even when the ELB has one bound. On this cluster that does not turn into a wrong public A-record: the external-dns Deployment in the external-dns namespace runs with --exclude-target-net=10.0.0.0/8, --exclude-target-net=172.16.0.0/12, --exclude-target-net=192.168.0.0/16 (plus the IPv6 equivalents fc00::/7 and fe80::/10) — verified with kubectl -n external-dns get deploy external-dns -o jsonpath='{.spec.template.spec.containers[0].args}'. 172.16.0.19 falls inside 172.16.0.0/12, so external-dns discards the target and creates no record at all — it does not write a bad one.

The end result looks the same either way — the service is unreachable from outside the VPC — but the failure mode, and where you go looking, are different. Search DNS for a wrong IP here and you find nothing, because no record was ever created; that absence can read as "external-dns is broken" and send you chasing the wrong system. The actual decision is made — and logged — by external-dns itself: check kubectl -n external-dns logs -l app.kubernetes.io/name=external-dns and the --exclude-target-net arguments above, not the DNS zone.

This protection is a configuration choice on this cluster, not a property of external-dns in general. Without those --exclude-target-net ranges configured, external-dns would write the private VIP straight into a public A-record — DNS would resolve, the record would show green, and the service would still be unreachable, but for the original reason: a wrong record, not a missing one. Check a cluster's own external-dns args before assuming either behaviour.

The tempting fix — setting kubernetes.io/elb.eip-id by hand — was tried and measured to do nothing: the annotation reaches the Service, the CCM reconciles, and EXTERNAL-IP stays the private VIP regardless. The only verified way out is setting external-dns.alpha.kubernetes.io/target to the EIP by hand on the Gateway. That target-annotation route has not itself been run end-to-end in this spike — it is the one solution the measurements point to, not yet a proven recipe.

3. The annotations that matter

The OTC CCM honours these kubernetes.io/elb.* annotations. Without elb.autocreate (or elb.id), the CCM skips the Service with "service annotation… is not defined, skip".

Annotation	Role	Field	Belegstatus
`kubernetes.io/elb.class`	`"union"` = shared ELB. Live-verified on all eight production Gateways. `"performance"` = dedicated ELB tier — no such Gateway currently exists on this cluster, so that half of the claim is unconfirmed here.	required	aus der CCE-Doku, hier nicht geprueft
`kubernetes.io/elb.autocreate`	JSON blob describing the ELB + EIP to create. Without it the CCM skips. The strict-schema behaviour (`bandwidth_name` required, `elb autoCreate field:[X] is invalid` on a missing field) is documented CCE behaviour that was not re-triggered on this cluster.	required for autocreate	aus der CCE-Doku, hier nicht geprueft
`kubernetes.io/elb.id`	Use an EXISTING ELB instead of creating one — but only if that ELB's VPC matches the node subnet's VPC (see the precondition box above). Mutually exclusive with `autocreate` by design; that exclusivity itself was not tested (both were never set together). The CCM does write `elb.id` back after autocreate — confirmed live on all eight Gateways today.	alternative	auf v2 gemessen (2026-08-26)
`kubernetes.io/elb.eip-id`	On the autocreate path, written back by the CCM and non-empty on all eight production Gateways. On the elb.id path this is not reliable: it came back empty even though the ELB had a bound EIP, and setting it by hand changed nothing in `EXTERNAL-IP`. Do not use this annotation to fix the external-dns problem below.	written back (autocreate only)	auf v2 gemessen (2026-08-26)
`kubernetes.io/elb.pass-through`	When `onlyLocal`, ELB → pod traffic bypasses kube-proxy SNAT. Envoy Gateway sets this by default — confirmed on all eight production Gateways.	optional	auf v2 gemessen (2026-08-26)

4. Verify it actually worked

There is no packaged demo script for either pattern in this repo today — the commands below are the real ones used to verify both patterns on this cluster on 2026-08-26. Pattern A is reproducible any time (it's how all eight production Gateways got their address); Pattern B was run in an isolated elbid-spike namespace, not against a production Service.

# Watch EXTERNAL-IP appear
kubectl -n <ns> get svc <name> -w

# The event stream is the real proof, not the field above — see why below
kubectl -n <ns> describe svc <name> | sed -n '/^Events:/,$p'

The deceptive part: success and failure look identical for two events

Both a working setup and a broken one (VPC mismatch, see the precondition box in section 2) go through the exact same first two events, and both end up with an EXTERNAL-IP set. Reading only kubectl get svc, or stopping at EnsuredLoadBalancer, is not enough evidence of success on this path.

	Working (same VPC)	Broken (VPC mismatch)
Event 1	`Normal EnsuringLoadBalancer`	`Normal EnsuringLoadBalancer`
Event 2	`Normal EnsuredLoadBalancer`	`Normal EnsuredLoadBalancer`
Event 3 (decisive)	✓ `Normal UpdatedLoadBalancer`	✗ `Warning UpdateLoadBalancerFailed`
`EXTERNAL-IP` set?	Yes — `172.16.0.19`	Yes — `10.43.0.77` (measured, but unreachable)
Traffic works?	Yes	No

The only reliable signal is the third event: UpdatedLoadBalancer means the node member was actually registered against the ELB; UpdateLoadBalancerFailed means it wasn't, no matter what EXTERNAL-IP shows.

Teardown: the ELB outlives the namespace

Deleting the Service and its namespace does not delete a hand-attached (elb.id) ELB — measured on this cluster: re-binding a fresh Service to the same elb.id after a full namespace deletion reported EnsuredLoadBalancer within ~51s and the identical VIP came back. The CCM does not own the lifecycle of an ELB it didn't create.

Separately: after deleting a Gateway, the corresponding Service in envoy-gateway-system can take a few seconds to disappear — asynchronous garbage collection, not a leak. Checking immediately and seeing it still there is not a sign anything is stuck; give it a moment before reaching for --force.

5. Direct LB vs Gateway API path

Both end up with an OTC ELB + EIP. The difference is who creates the Service and what sits between the ELB and your pod.

Direct LoadBalancer Service

You write the Service yourself. Traffic path:

OTC ELB :80
kube-proxy NodePort
Pod

Good for: one app, one IP, no L7 routing logic.

Limits: one ELB per Service. No path-/host-/header-based routing. No traffic splitting.

Gateway API path

You write a Gateway + HTTPRoute. The Gateway controller (Envoy Gateway) internally creates a Service of type LoadBalancer to expose itself. Traffic path:

OTC ELB :80
kube-proxy NodePort
Envoy data-plane pod
HTTPRoute match → backend pod

Good for: many apps sharing one ELB, path-/host-based routing, canary, header filters, gRPC, TLS termination.

The ELB mechanism is identical — Envoy Gateway just automates the step you took manually here.

1. What happens when you kubectl apply