Configure Load balancer and ingress controller to route application traffic to kubernetes worker nodes.

Configure HAProxy to handle application traffic on https and configure ingress controller to route the traffic to kubernetes worker nodes

Goal is to configure an external(external to kubernetes cluster) load balancer to accept application traffic on secure port and route them to applicable kubernetes worker node. We will use haproxy and nginx-ingress-controller here to achieve the goal. However same could be achieved using any other reverse proxy software or using any cloud load balancer in case you have access to them. We will also configure a FQDN at ingress end. FQDN must be pointed to virtual/floating IP of the load balancer in this case.

Prerequisite: Considering you already have haproxy installed. My recommendation would be to have high available haproxy installation coupled with keepalive and using VIP/floating ip. However we could consider single haproxy instance also. It would be single point of failure in that case(not recommended for production).

If you do not have haproxy installed then please follow the below link to install one

IP FailOver Setup Using KeepAlived on CentOS & Red Hat
Keepalived is used for IP failover between two servers. It facilities for load balancing and high-availability to…tecadmin.net

We would be doing below steps

Generate SSL certificate for the proxy.
Configure a https frontend and http backend balancing kubernetes worker.
Configure nginx-ingress controller in kubernetes cluster.
Create a ingress-controller entry for your application.
DNS entry update(or host file update)
Test the setup by deploying a test pod.

Step 1: Generate a SSL certificate for the proxy.

We will use openssl to generate self signed certificate. Please skip this if you already have one.

sudo mkdir -p /etc/ssl/openDrApp

sudo openssl genrsa -out /etc/ssl/openDrApp/openDrApp.key 1024

sudo openssl req -new -key /etc/ssl/openDrApp/openDrApp.key -out /etc/ssl/openDrApp/openDrApp.csr

#sign
sudo openssl x509 -req -days 365 -in /etc/ssl/openDrApp/openDrApp.csr -signkey /etc/ssl/openDrApp/openDrApp.key -out /etc/ssl/openDrApp/openDrApp.crt

#key + cert
sudo cat /etc/ssl/openDrApp/openDrApp.crt /etc/ssl/openDrApp/openDrApp.key | sudo tee /etc/ssl/openDrApp/openDrApp.pem

Step 2: Configure a https frontend and http backend balancing kubernetes worker.

In this step we would create a haproxy frontend entry to accept https entry, http backend entry to forward to kubernetes workers.

Define a frontend

frontend openDrApp_ssl_443
  bind 10.174.31.23:443 ssl crt /etc/ssl/openDrApp/openDrApp.pem
  mode http
  default_backend openDrApp_bk

Define a back-end to forward traffic to kubernetes workers

backend openDrApp_bk
  mode http
  balance roundrobin
  # server hostname ip check
  server opendrkwrkr001 10.0.1.11:80 check
  server opendrkwrkr002 10.0.1.12:80 check
  server opendrkwrkr003 10.0.1.13:80 check
  server opendrkwrkr004 10.0.1.14:80 check
  server opendrkwrkr005 10.0.1.15:80 check
  server opendrkwrkr006 10.0.1.16:80 check
  server opendrkwrkr007 10.0.1.17:80 check
  server opendrkwrkr008 10.0.1.18:80 check
  timeout connect        10s
  timeout server          1m

Restart the haproxy

sudo systemctl restart haproxy

#check status of proxy
sudo systemctl status haproxy -l

#add rule to firewall and reload
sudo firewall-cmd --zone=public --add-port=443/tcp --permanent

sudo firewall-cmd --zone=public --add-service=https --permanent
sudo firewall-cmd --reload

#check firewall rules
sudo firewall-cmd --zone=public --list-all

Step 3: Configure nginx-ingress controller in kubernetes cluster

In this step we would setup nginx-ingress controller for kubernetes cluster

Clone the artifacts

git clone https://github.com/nginxinc/kubernetes-ingress/
cd kubernetes-ingress/deployments
git checkout v1.8.0

Create kubernetes objects for ingress

kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f rbac/rbac.yaml
#ap-rbac for app protect only
kubectl apply -f rbac/ap-rbac.yaml
kubectl apply -f common/default-server-secret.yaml

kubectl apply -f common/nginx-config.yaml

#custom
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f common/ts-definition.yaml
kubectl apply -f common/policy-definition.yaml

#TCP/UDP traffic
kubectl apply -f common/gc-definition.yaml
kubectl apply -f common/global-configuration.yaml

#app protect
kubectl apply -f common/ap-logconf-definition.yaml 
kubectl apply -f common/ap-policy-definition.yaml

#deamon set
kubectl apply -f daemon-set/nginx-ingress.yaml

#ingress uninstall
#kubectl delete namespace nginx-ingress
#kubectl delete clusterrole nginx-ingress
#kubectl delete clusterrolebinding nginx-ingress

Check ingress-controller

kubectl -n nginx-ingress get all
# all pods should be in running state

Step 4: Create an ingress-controller entry for your application

We will create OpenDrAPP-ingress.yaml config to configure ingress-controller for the application

kubectl create ns opendrapp

#OpenDrApp ingress config
sudo cat > OpenDrAPP-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: opendrapp-ingress
spec:
  rules:
  - host: opendrapp.com
    http:
      paths:
      - path: /
        backend:
serviceName: opendrapp-ui
servicePort: 8080

EOF

#apply the config
kubectl -n opendrapp apply -f OpenDrAPP-ingress.yaml

#check the config
kubectl -n opendrapp get ing

Step 5: DNS update

Please update the DNS of domain

Domain name → load balancer VIP/floating ip

In this case, opendrapp.com →10.174.31.23

Or please update the client system host entry with the below entry

10.174.31.23 opendrapp.com

Please note that this configuration will not work without the DNS record update or host entry. Will fail at Ingress controller end.

Now, you should able access your application using below url

https://opendrapp.com

Step 5: Test

We will run a nginx test pod and cluster-ip service to test.

#nginx deployment
sudo cat > OpenDrAPP-ui-deployment.yaml <<EOF
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: opendrapp-ui
spec:
selector:
matchLabels:
app: opendrapp-ui
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: opendrapp-ui
spec:
containers:
      - name: nginx
image: nginx:1.14.2
ports:
        - containerPort: 80
EOF

#apply the deployment
kubectl -n opendrapp apply -f OpenDrAPP-ui-deployment.yaml

#inspect
kubectl -n opendrapp get pods

Create a cluster ip service top of the ui-app

sudo cat > OpenDrAPP-ui-svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
 name: opendrapp-ui-svc
 labels:
   app: opendrapp-ui
spec:
 ports:
 - port: 8080
   protocol: TCP
 selector:
   app: opendrapp-ui
EOF

#apply
 kubectl -n opendrapp apply -f OpenDrAPP-ui-svc.yaml
#Inspect
kubectl -n opendrapp get svc

you should able access https://opendrapp.com and see nginx welcome page now

Author

Md Shamimuddin(Shamim)

Solution Engineer with experience in Software architecture, Solution Design & Software Development, scrum master, conducting architectural evaluation, design and analysis of enterprise-wide system. Motivated leader who leads teams through all phases of software development life cycle.

View all posts

Configure Load balancer and ingress controller to route application traffic to kubernetes worker nodes.

Author

Leave a Reply