Trendy Ingress for AKS: Introducing Utility Gateway for Containers (AGC)

Hi there People!

This Microsoft Azure Infra Summit 2026 periods one is for everybody working manufacturing workloads on AKS. Jack Stromberg from the Utility Gateway product workforce walked us via your entire historical past of load balancing in about ten minutes (no small feat) after which dropped the information that Utility Gateway for Containers (AGC) is choosing up a model new trick: AI gateway capabilities via the CNCF Gateway API Inference Extension. When you have ever watched one pod get hammered whereas the others sit idle, this session goes to really feel private.

📺 Watch the session:

The NGINX Ingress Controller retirement is pushing quite a lot of groups to select a brand new entrance door for his or her AKS clusters. AGC is Microsoft’s managed, Azure-native reply. It’s Layer 7, it speaks each the Kubernetes Ingress API and the newer Gateway API, and it runs outdoors the cluster so you aren’t paying with pod CPU to terminate TLS.

Right here is the brief checklist of why this session issues:

• The web runs on load balancers, and the load balancer story has been principally steady for many years. That’s altering quick.
• AI inference site visitors doesn’t seem like internet site visitors. Lengthy-lived, GPU-bound, costly. Conventional Layer 7 guidelines can’t sustain.
• AGC now ships AI gateway capabilities (in preview) that route by mannequin, route by load, and cease slamming a busy pod when others are idle.
• Measured numbers from the demo: zero timeouts versus 15 failed requests, and a P99 time-to-first-token of about 1,000 ms versus 16,000 ms on a vanilla HTTPRoute.

In case your AKS clusters are beginning to host self-hosted fashions alongside common providers, you might want to know this exists.

AGC is the evolution of the older Utility Gateway Ingress Controller (AGIC). The structure is the half that issues:

• Utility Gateway for Containers useful resource: the managed Azure useful resource that truly handles site visitors, TLS, WAF, and routing.
• Frontends: listeners and IP addresses that settle for incoming site visitors.
• Associations: the hyperlink between the AGC useful resource and your AKS VNet/subnet.
• ALB Controller: a small controller that runs inside your AKS cluster, watches your Gateway API and Ingress assets, and reconciles the AGC configuration in close to actual time.

Two deployment fashions. You’ll be able to let the ALB Controller handle the lifecycle of the AGC useful resource for you, or you may go Deliver Your Personal (BYO) and handle it with Bicep, Terraform, ARM, or the CLI. I like BYO for manufacturing as a result of it provides the platform workforce a clear separation between Kubernetes intent and the Azure useful resource lifecycle.

Characteristic-wise it covers what you’d count on from a contemporary Layer 7: automated retries, autoscaling, zone resiliency, weighted/break up site visitors for blue-green, header and URL rewrites, WAF integration, mTLS (entrance, again, or end-to-end), HTTP/2, WebSocket, gRPC, and server-sent occasions. Routing by hostname, path, header, question string, or HTTP methodology.

Jack opened with a tour of the OSI mannequin that I genuinely loved. The TL;DR:

• Layer 4 (community load balancer): stateless, quick, easy. Supply IP, vacation spot IP, ship. This nonetheless powers many of the web.
• Layer 7: TLS, session affinity, WAF, mutual auth, clever routing based mostly on the contents of the request. That is the place trendy internet functions dwell.
• Kubernetes ingress: the Layer 7 load balancer is not a {hardware} equipment you rack and stack. Pods come and go always. You want a controller that interprets Kubernetes intent into load balancer config in close to actual time.
• Gateway API: the CNCF’s vendor-neutral solution to categorical that intent. AGC is a first-class Gateway API implementation, so the YAML you write is moveable throughout clouds and on-prem.

In Gateway API land you’re employed with three major assets:

1. A Gateway (the entrance door, together with listeners and hostnames).
2. An HTTPRoute that claims “for requests matching this host/path/header, ship them to this backend.”
3. A backend, historically a Kubernetes Service.

That’s the image for regular internet apps. AI workloads break it.

Right here is the sincere tradeoff Jack laid out. Conventional Layer 7 load balancers had been tuned for short-lived, low-latency requests. They distribute site visitors with easy algorithms (spherical robin, least connections) as a result of they assume backends are interchangeable.

AI inference flips each a type of assumptions:

• Requests are long-lived. A mannequin reasoning over an enormous immediate can take many seconds.
• Backends will not be interchangeable. Pod A could be working a picture mannequin, Pod B a coding mannequin.
• Compute is pricey. GPUs will not be low cost, and a sizzling pod with a full KV cache can’t simply settle for extra work.
• A conventional load balancer will fortunately pile new requests on a pod that’s already saturated, whereas idle pods sit ready.

The repair is the CNCF Gateway API Inference Extension. It introduces 4 primitives, and AGC implements them so that you should not have to bolt the items collectively your self:

• Physique-Primarily based Router (BBR): reads the immediate physique to tug out the requested mannequin. In AGC, this runs contained in the load balancer itself. Nothing additional so that you can deploy.
• Endpoint Picker (EPP): runs in your cluster, scrapes metrics out of your mannequin pods (issues like KV cache utilization), and tells the load balancer which pod is definitely a very good goal proper now.
• InferencePool: a brand new Kubernetes useful resource that teams pods serving the identical mannequin together with the EPP and goal port.
• InferenceObjective: enables you to categorical precedence when a number of requests are competing for a similar pool.

The neat half is that InferencePool plugs in as a backendRef on an HTTPRoute, proper subsequent to conventional Service backends. You’ll be able to combine AI and customary workloads behind the identical gateway. No new gateway, no new product, simply an extra backend kind.

Jack ran two demos that I need to name out:

• Mannequin-aware routing: a script fires curl requests with completely different prompts. AGC’s body-based router pulls the requested mannequin out of the JSON physique, units a header, the HTTPRoute matches the header, and site visitors lands on the proper InferencePool. The vanilla HTTPRoute facet can’t do that and quite a lot of requests fail.
• Load-aware routing: three pods, one already busy. Hearth 50 requests on the AI-enabled route and 50 on the conventional route. The endpoint picker steers site visitors to the idle pods. P99 time-to-first-token: about 1,000 ms with AGC’s AI gateway, about 16,000 ms with out. Timeouts: zero with AGC, 15 with out.

That may be a actual quantity, on the identical backend, in the identical cluster. The one factor that modified is the load balancer realizing what is occurring behind it.

If you wish to do this, right here is the trail I might take:

• Rise up an AKS cluster with Azure CNI, OIDC issuer, and Workload Identification enabled. AGC wants these.
• Register the suppliers: Microsoft.ContainerService, Microsoft.Community, Microsoft.NetworkFunction, Microsoft.ServiceNetworking.
• Create a user-assigned managed identification for the ALB Controller and grant it the AGC Configuration Supervisor function.
• Set up the ALB Controller through Helm or as an AKS add-on, then create your Gateway useful resource towards the azure-alb-external GatewayClass.
• Begin with a plain HTTPRoute towards an everyday Kubernetes Service. Get that inexperienced first.
• If you find yourself prepared for inference, deploy your mannequin, drop in an InferencePool and an Endpoint Picker, and level your HTTPRoute on the pool.

Pierre’s tip: the AI gateway capabilities are in preview. The workforce explicitly requested for suggestions from anybody working self-hosted fashions in AKS, particularly round mannequin combine, request patterns, and what you might be doing right now to maintain sizzling pods from getting buried.

Catch each session from the Microsoft Azure Infra Summit 2026 on the official playlist: https://www.youtube.com/playlist?checklist=PLjt5SKzX1iI8con7FJDB56G6hHqxGm7ki. There’s a companion session on AKS Safety with AGC and Managed Cilium that pairs very well with this one.

Cheers!

Pierre Roman