Migrating from Ingress to API Gateway in Kubernetes

The Evolution of Traffic Management in Kubernetes

For years, the Ingress resource has been the standard way to expose services in a Kubernetes cluster. It was simple, effective, and served its purpose well in the early days of cloud-native development. However, as architectures became more complex-with multi-cluster setups, fine-grained traffic splitting, and advanced security requirements-the limitations of the standard Ingress resource began to show.

Enter the Kubernetes Gateway API. This is not just a replacement but a complete evolution of how we handle North-South traffic. It provides a more expressive, extensible, and role-oriented set of resources to manage traffic.

Why Migrate?

The standard Ingress resource is quite limited. To achieve anything beyond basic path-based routing, developers often had to rely on annotations specific to their Ingress controller (like Nginx). This led to "annotation hell," where your configuration was non-portable and difficult to maintain.

Key Benefits of API Gateway (Gateway API):

• Role-Oriented: It separates infrastructure concerns (GatewayClass, Gateway) from application routing concerns (HTTPRoute), allowing infra teams and app teams to work independently.
• Expressiveness: Standard support for header-based routing, traffic splitting (canary releases), and redirects without custom annotations.
• Extensibility: Built to be extended with custom resources (Policy attachment) for specific vendor features while keeping the core logic standardized.
• Portability: Move between different implementations (Kong, Traefik, Istio) without rewriting your entire configuration, as the core resources are standardized.

Quick Comparison: Ingress vs. Gateway API

How to Migrate: The Transition Path

Migrating doesn't mean you have to rip and replace everything overnight. The Gateway API can coexist with your existing Ingress controllers. The transition typically follows these steps:

1. Install a Gateway API Controller: Most modern controllers (Traefik v3, Kong, Istio) now support both.
2. Define a GatewayClass and Gateway: This represents the entry point (the "Load Balancer").

# Defining the entry point
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
spec:
  gatewayClassName: my-gateway-class
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same

3. Create HTTPRoutes: Translate your Ingress rules into HTTPRoute resources. This is where you map paths and hosts to services.

# Mapping traffic to your service
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - "app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: my-api-service
      port: 80

4. Gradual Traffic Shift: Use your DNS or an external load balancer to slowly shift traffic from the Ingress IP to the new Gateway IP.

From Ingress to HTTPRoute: A Concrete Example

To understand the shift, let's look at how a standard Ingress manifest translates to the Gateway API. Notice how the configuration is cleaner and avoids controller-specific annotations.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: api.neneos.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "api.neneos.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefix: /
    backendRefs:
    - name: api-service
      port: 80

Advanced Configuration & Missing Pieces

1. Cross-Namespace Routing (ReferenceGrant)

One of the most powerful features of the Gateway API is the ability for a Gateway in one namespace to route traffic to a Service in another namespace. To prevent security risks, the Gateway API introduces the ReferenceGrant resource. This must be created in the target namespace (where the Service lives) to explicitly allow the Gateway to reference it.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gateway-to-service
  namespace: prod-apps
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: infrastructure
  to:
  - group: ""
    kind: Service

2. TLS/SSL Certificate Management

In the Gateway API, TLS is configured at the Gateway level (the listener). If you use cert-manager, the integration is seamless via annotations on the Gateway resource, much like you did with Ingress.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: my-gateway-class
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: neneos-com-tls

3. Replacing Annotations with Filters

Standard features like CORS, Rate Limiting, and Header manipulation are now handled by Filters within the HTTPRoute. This eliminates vendor-specific annotation hell. Most modern controllers like Kong or Traefik implement these filters natively as part of the spec.

Here is an example of common filter patterns for header manipulation and redirects:

# Header Manipulation & Redirect Example
rules:
- matches:
  - path: { type: PathPrefix, value: /old-api }
  filters:
  - type: RequestHeaderModifier
    requestHeaderModifier:
      add:
      - name: x-gateway-source
        value: "k8s-gateway"
      remove: ["x-internal-token"]
  - type: RequestRedirect
    requestRedirect:
      scheme: https
      statusCode: 301

4. End-to-End Encryption (BackendTLSPolicy)

While Ingress often handled TLS termination at the edge, the Gateway API introduces BackendTLSPolicy for secure communication between the Gateway and the backend service. This allows for full end-to-end encryption using custom CAs or provided certificates, satisfying strict compliance requirements without needing a full service mesh in some cases.

Observability and Operations

For platform operators, the Gateway API provides clear status reporting. Each resource has a status block that shows why a route is failing, such as a hostname conflict, a missing backend, or a configuration error. Most Gateway API implementations (such as Istio or Envoy Gateway) provide direct integration with Prometheus and Grafana for metrics on throughput, latency, and error rates.

Rollout Strategy

When switching, we recommend a parallel deployment:

• Keep your Ingress controller running.
• Deploy the Gateway API controller and your HTTPRoutes.
• Test the new Gateway using its dedicated LoadBalancer IP or a temporary DNS entry.
• Once validated, update your production DNS records. Keep in mind the DNS TTL; lower it to 60-300 seconds before the switch to allow for quick rollbacks.

Possible Difficulties & Best Practices

Every migration comes with challenges. When moving to the Gateway API, watch out for:

• Learning Curve: There are more resource types to manage (Gateway, HTTPRoute, ReferenceGrant) compared to a single Ingress file. Start with a small, non-critical service to get used to the hierarchy.
• Version Compatibility: Ensure your Kubernetes version (1.24+) and your controller version fully support the Gateway API specs. Check if the controller supports the 'Standard' or 'Experimental' channel of the API.
• CRD Installation: Unlike Ingress, the Gateway API CRDs are often not installed by default in older clusters. You may need to install them manually using the official manifests.
• Feature Parity: Some highly specialized Nginx modules or Lua scripts might not have a direct 1:1 filter equivalent yet. In these cases, look for vendor-specific ExtensionRef filters.

Best Practices for a Smooth Move:

• Automate Translation: Use tools like ingress2gateway (a SIG-Network project) to generate your initial HTTPRoute manifests from existing Ingress resources.
• Namespace Strategy: Keep your Gateway resources in a dedicated infrastructure namespace and your HTTPRoutes in the same namespace as your applications.
• Validate with Admission Webhooks: Ensure your Gateway controller has its admission webhook enabled to catch configuration errors before they are applied.

"The Gateway API represents the most significant improvement to Kubernetes networking since its inception, moving from 'good enough' to 'production-grade' traffic management."

The Old Guard: Nginx Ingress

NGINX Ingress Controller is the most widely used ingress solution today. It is robust and battle-tested. However, its heavy reliance on annotations to implement Layer 7 features is exactly what the Gateway API aims to solve.

# Typical Ingress with Nginx annotations
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

While Nginx now has support for the Gateway API, many organizations are taking this migration as an opportunity to look at more modern, native solutions.

The Top 3 API Gateway Solutions

If you are looking to move beyond standard Ingress, these are the three most popular solutions in the ecosystem right now:

1. Kong: A veteran in the API Gateway space. Kong for Kubernetes offers incredible performance and a vast plugin ecosystem for authentication, rate limiting, and observability.
2. Istio: While primarily a service mesh, Istio's Gateway implementation is one of the most mature. It's the go-to choice if you need deep security and observability across your entire microservices landscape.
3. Traefik: Known for its simplicity and dynamic configuration. Traefik v3 has first-class support for Gateway API and is exceptionally easy to set up with modern CI/CD pipelines.

Azure AKS and the Gateway API

For those running on Azure Kubernetes Service (AKS), Microsoft has been heavily investing in this space. Traditionally, AGIC (Application Gateway Ingress Controller) was the standard for integrating with Azure Application Gateway.

Today, Azure offers the Application Gateway for Containers (ALB), which is a fully managed service that implements the Kubernetes Gateway API. This provides a high-performance, scalable entry point that is managed by Azure but controlled entirely through standard Kubernetes manifests. It's the recommended path forward for any new AKS deployments looking for native Azure integration with modern standards.