Setting up ExternalDNS using the same domain for public and private Route53 zones

This tutorial describes how to setup ExternalDNS using the same domain for public and private Route53 zones and nginx-ingress-controller. It also outlines how to use cert-manager to automatically issue SSL certificates from Let's Encrypt for both public and private records.

Deploy public nginx-ingress-controller

Consult External DNS nginx ingress docs for installation guidelines.

Specify ingress-class in nginx-ingress-controller container args:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: external-ingress
  name: external-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-ingress
  template:
    metadata:
      labels:
        app: external-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/external-ingress-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/external-tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/external-udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=external-ingress
        - --publish-service=$(POD_NAMESPACE)/external-ingress
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: external-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Set type: LoadBalancer in your public nginx-ingress-controller Service definition.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: external-ingress
  name: external-ingress
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: external-ingress
  sessionAffinity: None
  type: LoadBalancer

Deploy private nginx-ingress-controller

Consult External DNS nginx ingress docs for installation guidelines.

Make sure to specify ingress-class in nginx-ingress-controller container args:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: internal-ingress
  name: internal-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: internal-ingress
  template:
    metadata:
      labels:
        app: internal-ingress
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/internal-ingress-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/internal-tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/internal-udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --ingress-class=internal-ingress
        - --publish-service=$(POD_NAMESPACE)/internal-ingress
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: internal-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Set additional annotations in your private nginx-ingress-controller Service definition to create an internal load balancer.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: internal-ingress
  name: internal-ingress
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: internal-ingress
  sessionAffinity: None
  type: LoadBalancer

Deploy the public zone ExternalDNS

Consult AWS ExternalDNS setup docs for installation guidelines.

In ExternalDNS containers args, make sure to specify aws-zone-type and either ingress-class or annotation-filter (depending on whether your cluster makes use of ingressClassName):

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    app: external-dns-public
  name: external-dns-public
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns-public
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns-public
    spec:
      containers:
      - args:
        - --source=ingress
        - --provider=aws
        - --registry=txt
        - --txt-owner-id=external-dns
        - --ingress-class=external-ingress
        # ... or, if you use annotations for ingress classes
        # - --annotation-filter=kubernetes.io/ingress.class in (external-ingress)
        - --aws-zone-type=public
        image: registry.k8s.io/external-dns/external-dns:v0.13.4
        name: external-dns-public

Deploy the private zone ExternalDNS

Consult AWS ExternalDNS setup docs for installation guidelines.

In ExternalDNS containers args, make sure to specify aws-zone-type and either ingress-class or annotation-filter (depending on whether your cluster makes use of ingressClassName):

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    app: external-dns-private
  name: external-dns-private
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns-private
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns-private
    spec:
      containers:
      - args:
        - --source=ingress
        - --provider=aws
        - --registry=txt
        - --txt-owner-id=dev.k8s.nexus
        - --ingress-class=internal-ingress
        # ... or, if you use annotations for ingress classes
        # - --annotation-filter=kubernetes.io/ingress.class in (internal-ingress)
        - --aws-zone-type=private
        image: registry.k8s.io/external-dns/external-dns:v0.13.4
        name: external-dns-private

Create application Service definitions

For this setup to work, you need to create two Ingress definitions for your application.

At first, create public Ingress definition (make sure to un-comment either the annotations or ingressClassName lines):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  # uncomment if you use annotations for ingress classes
  # annotations:
  #   kubernetes.io/ingress.class: "external-ingress"
  labels:
    app: app
  name: app-public
spec:
  # uncomment if you use ingressClassName
  # ingressClassName: external-ingress
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix

Then create private Ingress definition (again, make sure to un-comment either the annotations or ingressClassName lines):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  # uncomment if you use annotations for ingress classes
  # annotations:
  #   kubernetes.io/ingress.class: "internal-ingress"
  labels:
    app: app
  name: app-private
spec:
  # uncomment if you use ingressClassName
  # ingressClassName: internal-ingress
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix

Additionally, you may leverage cert-manager to automatically issue SSL certificates from Let's Encrypt. To do that, request a certificate in public service definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-challenge-type: "dns01"
    certmanager.k8s.io/acme-dns01-provider: "route53"
    certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
    kubernetes.io/tls-acme: "true"
  labels:
    app: app
  name: app-public
spec:
  ingressClassName: "external-ingress"
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix
  tls:
  - hosts:
    - app.domain.com
    secretName: app-tls

And reuse the requested certificate in private Service definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: app
  name: app-private
spec:
  ingressClassName: "internal-ingress"
  rules:
  - host: app.domain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        pathType: Prefix
  tls:
  - hosts:
    - app.domain.com
    secretName: app-tls

11 KiB Raw Blame History