Configure and Manage a Kubernetes HAProxy Ingress Controller
Table of Contents
- Exposing services to the world
- Introducing Ingress
- Introducing Ingress Controllers
Almost everyone who is deploying an application would like the app to be accessible to other people on the public internet.
- If you are an independent developer working with a cloud provider, then you'll ask, what's my public IP address, and what ports are exposed?
- If you are a developer in a corporation with private data center resources, you'll ask, how do I get the load-balancer rules configured for my set of internal hosts (and who do I need to talk to)?
- If you are working an application deployed inside a Kubernetes cluster, you'll ask, what's going on here at all? How do I let people get in?
If you need to expose your Kubernetes services to the world, Ingresses are the way to go. At the time of this writing Ingress is only available in beta, so let's see the alternatives first.
Exposing services to the world
Your application in a Kubernetes cluster probably contains these items:
Your carefully-designed set of applications has already been put into a set of containers-sharing-resources as a Kubernetes pod, so how do those processes communicate with others?
The pod exposes endpoints usually at the overlay network. That network is not reachable from outside the cluster. You can also use
hostPort to expose the endpoint at the host IP, or set
hostNetwork: true to use the host's network interface from your pod. But in both scenarios you should take extra care to avoid port conflicts at the host, and possibly some issues with packet routing and name resolutions. So let's assume that we want a risk-free and scalable kubernetes cluster, and we are properly using pods exposing their functions to the container ports.
Kubernetes services primarily work to interconnect different pod-based applications. They're designed to help make a microservices system scale well internally. They are not primarily intended for external access, but there are some accepted ways to expose services to external clients.
Let's simplify services as a routing, balancing and discovery mechanism for the pod's endpoints. Services target pods using selectors, and can map container ports to service ports. A service exposes one or more ports, although usually you will find that only one is defined.
A service can be exposed using 3
ClusterIP: The default, only exposed within the cluster.
NodePort: Uses a port from a cluster defined range (usually 30000-32767) at every cluster node to expose the service.
LoadBalancer: Setting your cluster kubelet with the parameter
--cloud-provider=the-cloud-providerbrings some goodies, including the ability to use the provider's load balancer.
If we choose
NodePort to expose our services, we will still need an external proxy that uses DNAT to expose more friendly ports. That element will also need to balance on the cluster nodes, which in turn will balance again through the service to the internal pods. There is no easy way of adding TLS or simple host header routing rules to the external service.
LoadBalancer is probably the easiest of all methods to get your service exposed to the internet. The problem is that there is no standard way of telling a Kubernetes service about the elements that a balancer requires, again TLS and host headers are left out.
Endpoints are usually automatically created by services, unless you are using headless services and adding the endpoints manually. An endpoint is a host:port tuple registered at Kubernetes, and in the service context it is used to route traffic. The service tracks the endpoints as pods that match the selector are created, deleted and modified. Individually, endpoints are not useful to expose services, since they are to some extent ephemeral objects.
- If you can rely on your cloud provider to correctly implement the LoadBalancer for their API, to keep up-to-date with Kubernetes releases, and you are happy with their management interfaces for DNS and certificates, then setting up your services as type
LoadBalanceris quite acceptable. On the other hand, if you want to manage load balancing systems manually and set up port mappings yourself,
NodePortis a low-complexity solution. If you are directly using
Endpointsto expose external traffic, perhaps you already know what you are doing (but consider that you might have made a mistake, there could be another option).
Given that none of these elements has been originally designed to expose services to the internet, their functionality may seem limited for this purpose.
The Ingress resource is a set of rules that map to Kubernetes services. Sure, there will be a more complex definition, and I'll be missing some points, but I still think that "rules to services mapping" is the best way to understand what an Ingress does.
Ingress resources are defined purely within Kubernetes as a object that other entities can watch and respond to.
Supported rules at this beta stage are:
- host header: So we can forward traffic based on domain names.
- paths: Looks for a match at the the beginning of the path.
TLS: If the ingress adds TLS, HTTPS and a certificate configured through a secret will be used.
When no host header rules are included at an Ingress, requests without a match will use that Ingress and be mapped to the backend service. You will usually do this to send a 404 page to requests for sites/paths which are not sent to the other services.
Ingress tries to match requests to rules, and forwards them to backends, which are composed of a service and a port (remember that a service can contain multiple ports.
To summarize: An Ingress defines how to take a request and (based on host/path/tls) send it to a backend.
Introducing Ingress Controllers
In order to grant (or remove) access, some entity must be watching and responding to changes in the services, pods and Ingresses. That entity is the Ingress controller. While the Ingress controller does work directly with the Kubernetes API, watching for state changes, it is not coupled as closely to the Kubernetes source code as the cloud-provider LoadBalancer implementations.
Ingress controllers are applications that watch Ingresses in the cluster and configure a balancer to apply those rules. Typically you will use an existing Ingress controller that controls a third party balancer like HAProxy, NGINX, Vulcand or Traefik, updating configuration as the Ingress and their underlying elements change.
Ingress controllers will usually track and communicate with endpoints behind services instead of using services directly. This way some network plumbing is avoided, and we can also manage the balancing strategy from the balancer.
Ingress Controller extensions
As of this writing, Ingress doesn't support TCP balancing, balancing policies, rewriting, SNI, and many other common balancer configuration parameters. Ingress controller developers have extended the Ingress definition using annotations, but be aware that those annotations are bound to the controller implementation. This means that the Ingress resource will evolve, and might make some of these annotations obsolete in the future.
This example will be a demonstration of Ingress usage. For the demo, we will create in our cluster:
- A backend that will receive requests for
- A pair of backends that will receive request for
- One whose path begins with
- One whose path begins with
- One whose path begins with
- A default backend that shows a 404 page
You can use any website or service in a container as a backend. We will create three backends.
echoheaders: pod/service which contains a webpage that shows information about the received request.
--- apiVersion: v1 kind: ReplicationController metadata: name: echoheaders spec: replicas: 1 template: metadata: labels: app: echoheaders spec: containers: - name: echoheaders image: gcr.io/google_containers/echoserver:1.4 ports: - containerPort: 8080 readinessProbe: httpGet: path: /healthz port: 8080 periodSeconds: 1 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 10 --- apiVersion: v1 kind: Service metadata: name: echoheaders labels: app: echoheaders spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: echoheaders
default-http-backend: pod/service which is a simple 404 static page.
--- apiVersion: v1 kind: ReplicationController metadata: name: default-http-backend spec: replicas: 2 selector: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: gcr.io/google_containers/defaultbackend:1.0 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: default-http-backend
game2048: A pod/service which contains a static web page game
--- apiVersion: v1 kind: ReplicationController metadata: name: game2048 labels: name: game2048 spec: replicas: 2 selector: name: game2048 template: metadata: labels: name: game2048 version: stable spec: containers: - name: game2048 image: alexwhen/docker-2048 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: game2048 labels: name: game2048 spec: ports: - port: 80 targetPort: 80 selector: name: game2048
Create the backends of your choice with
kubectl create -f ...
For this example we will be using standard functionality only, so you should be able to use any Ingress controller. I'm using the HAProxy Ingress controlle.
The HAProxy Ingress controller is already configured and listening to changes in your Kubernetes cluster. If you prefer to get NGINX, it can also be deployed in any cluster following these instructions
Let's start by creating the
game2048 service at the balancer.
--- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: game-ingress spec: rules: - host: game.domain1.io http: paths: - path: backend: serviceName: game2048 servicePort: 80
That's it. We need now to reach HAPRoxy node using the host header
- Use your own domain, and point the A record to the HAProxy node's IP address.
curl -H "Host: game.domain1.io" real.server.address
- Install a browser plugin to add host header, like Virtual-Hosts for Chrome.
If you use a browser, you should be shown the 2048 game. But if you try the IP address without that host header, you wil get a 503 error.
We'll be fixing that with our next Ingress, the default backend:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: default-http-backend spec: backend: serviceName: default-http-backend servicePort: 80
Every request that doesn't match an existing rule will be served by the
default-http-backend. Now if you try the IP address without the game host header, you should get a simple 404 page.
Finally, let's use the flexible
echoheaders service to add some path matching:
--- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: echoheaders spec: rules: - host: domain2.io http: paths: - path: /path1 backend: serviceName: echoheaders servicePort: 80 - path: /path2 backend: serviceName: echoheaders servicePort: 80
In a real world scenario you will use different services for each path, but for us to test it's ok to use the echoheaders service.
To test all Ingresses try these requests:
- balancer node IP --> will be sent to
- domain1.io --> will be sent to
- domain1.io/path1 --> will be sent to
game2048(and it will fail with 404, since game2048 has no resource at that path)
- domain2.io/path1 --> will be sent to
- domain2.io/path2 --> will be sent to
- domain2.io/path3 --> will be sent to
- domain2.io --> will be sent to
Ingresses are simple and very easy to deploy, and really fun to play with. However, when you plan your Ingresses for production some other factors arise:
- High Availability
- Custom configuration
But we will leave those questions for a future post.