Kubernetes Networking for Multi-Tenant Clusters: Isolation and Safety

The Multi-Tenant Dilemma: Sharing Without Compromising

In many organizations, a single large Kubernetes cluster is shared among multiple teams. This "multi-tenant" approach is efficient for resource utilization, but it creates significant challenges for networking and security. How do you allow Team A to manage their own routes without giving them the power to accidentally (or intentionally) hijack Team B's traffic or break the shared entry point?

The traditional Ingress resource struggled here. Because Ingress often mixed infrastructure and application logic in a single resource, RBAC (Role-Based Access Control) became a blunt instrument. If you gave a team access to create an Ingress, they could often claim any hostname or path they wanted.

The Kubernetes Gateway API is "Role-Oriented" by design, specifically to solve these multi-tenant headaches.

Roles in the Gateway API

The API explicitly separates three main personas:

• Infrastructure Provider: Manages GatewayClass (the "How" of the load balancer).
• Cluster Operator (SRE/Platform Team): Manages the Gateway resource (the "Where" of the entry point). Usually sits in a restricted infrastructure namespace.
• Application Developer: Manages HTTPRoutes (the "What" of the routing). Sits in the team's specific namespace.

The Security Barrier: ReferenceGrant

A key security mechanism in the Gateway API is ReferenceGrant. In a multi-tenant cluster, you often have a Gateway in the infra namespace that needs to route traffic to a Service in the app-team-a namespace.

In the Ingress world, this cross-namespace reference was either impossible or required risky cluster-wide permissions. The Gateway API requires a handshake:

1. The Gateway (in infra) tries to reach a Service in app-team-a.
2. The App Team must explicitly "allow" this connection by creating a ReferenceGrant in their own namespace.

Example: Granting Access to the Shared Gateway

Here is how Team A allows the shared Gateway to "see" their services:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-shared-gateway
  namespace: app-team-a
spec:
  from: # Who is allowed to reference me?
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: infra
  to: # What resources are they allowed to see?
  - group: ""
    kind: Service

Hostname Lockdown: Preventing Host Hijacking

A common fear in multi-tenant clusters is "host hijacking," where Team B deploys a route for api.teama.com and successfully steals Team A's traffic. The Gateway API prevents this through AllowedRoutes on the Gateway resource.

The Platform Team can configure the Gateway to only accept routes from specific namespaces that match specific hostnames.

# In the Gateway resource (managed by Infra Team)
listeners:
- name: https
  hostname: "teama.com" # Lockdown to this domain
  allowedRoutes:
    namespaces:
      from: Selector
      selector:
        matchLabels:
          team: team-a

RBAC Best Practices

To implement this successfully, your RBAC should follow the principle of least privilege:

• App Teams: get, list, watch, create, update on HTTPRoutes and ReferenceGrants within their own namespace. Crucially: No permission to touch Gateway or GatewayClass.
• Platform Team: Full control over the infra namespace and the ability to define GatewayClasses at the cluster level.

The Result: Scalable Self-Service

With this setup, you achieve a "Self-Service" model. When Team A needs a new route, they simply deploy an HTTPRoute in their namespace. They don't need to open a ticket with the Platform Team, and they cannot accidentally affect Team B's routes or the global Gateway configuration. The isolation is enforced by the Kubernetes API itself, not just by convention.

"The Gateway API doesn't just manage traffic; it manages the relationships between the teams that build and run your applications."