(message "Blog")

Automatic DNS With external-dns, UniFi, and Cilium BGP

·5 min read

Every homelab reaches the point where you deploy a new service, it gets an IP, and then you trot off to the router UI to add yet another DNS record by hand. I did that for years. This is the setup that finally made me stop: Cilium announces LoadBalancer IPs to my UniFi gateway over BGP, and external-dns writes the DNS records into UniFi automatically. Deploy an HTTPRoute, get a routable IP and a resolvable name, touch nothing.

This post is intentionally short and mostly configuration. In the Migrating Away From GitHub article I already covered how the Cilium Gateway API entry point itself is set up, so I won’t repeat that here.

BGP between UniFi and Cilium

Cilium hands out LoadBalancer IPs from a pool, but those IPs live on no physical network. Something has to tell the router where they are, and that something is BGP. UniFi gateways run FRR under the hood, and since UniFi Network 9.x you can upload an FRR config directly in the UI under Settings, Routing, BGP.

Mine peers with every K8s node using a private ASN, iBGP style:

router bgp 64512
 bgp router-id 10.42.0.1
 bgp log-neighbor-changes

 neighbor 10.42.3.1 remote-as 64512
 neighbor 10.42.3.1 description "debian-3001"

 neighbor 10.42.3.2 remote-as 64512
 neighbor 10.42.3.2 description "debian-3002"

 ! ...one stanza per node...

 address-family ipv4 unicast
  neighbor 10.42.3.1 activate
  neighbor 10.42.3.2 activate
 exit-address-family

On the Cilium side, the BGP control plane needs to be enabled in the Helm values:

bgpControlPlane:
  enabled: true

Then a pool to allocate LoadBalancer IPs from:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "lb-pool-1"
spec:
  blocks:
    - cidr: "10.128.0.1/24"

And the three resources that make the announcing happen. The cluster config says who to peer with:

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: bgp-cluster-config
spec:
  nodeSelector:
    matchLabels:
      bgp-policy: a
  bgpInstances:
    - name: instance-64512
      localASN: 64512
      peers:
        - name: peer-router
          peerASN: 64512
          peerAddress: 10.42.0.1
          peerConfigRef:
            name: cilium-peer

The peer config sets timers and points at the advertisement:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  timers:
    connectRetryTimeSeconds: 120
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: bgp

And the advertisement itself says what to announce, in my case LoadBalancer IPs of all services. The NotIn selector with a value nobody will ever use is the standard trick for “match everything”, because the selector is mandatory:

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchExpressions:
          - key: somekey
            operator: NotIn
            values:
              - never-used-value

The nodes also need the label the cluster config selects on:

kubectl label node debian-3001 bgp-policy=a

Verification is one command:

$ cilium bgp peers
Node          Local AS   Peer AS   Peer Address   Session State   ...
debian-3001   64512      64512     10.42.0.1      established

From that moment, every LoadBalancer service in the cluster gets an IP from the pool that is reachable from anywhere on my network.

external-dns writing into UniFi

Routable IPs are only half of the job, I still want names. UniFi is already the DNS server for my LAN, so instead of running yet another DNS service in the cluster, external-dns writes records straight into it through the external-dns-unifi-webhook provider from the home-operations folks.

I deploy it with the official Helm chart through Argo CD, but the interesting part is just the values:

fullnameOverride: external-dns
# Publish HTTPRoute hostnames -> Cilium gateway LB IP into UniFi DNS.
# `crd` adds DNSEndpoint resources for records with no Gateway API source.
sources:
  - gateway-httproute
  - crd
domainFilters:
  - mydomain
# sync (not upsert-only) so deleting an HTTPRoute also removes its
# record; ownership TXT records limit deletion to records it created.
policy: sync
txtOwnerId: k8s
provider:
  name: webhook
  webhook:
    image:
      repository: ghcr.io/home-operations/external-dns-unifi-webhook
      tag: "0.10.9"
    env:
      - name: UNIFI_HOST
        value: https://10.42.0.1
      - name: UNIFI_API_KEY
        valueFrom:
          secretKeyRef:
            name: unifi-dns
            key: UNIFI_API_KEY

The API key comes from the UniFi console under Control Plane, Integrations, API keys, and lives in a plain secret:

apiVersion: v1
kind: Secret
metadata:
  name: unifi-dns
  namespace: external-dns
type: Opaque
stringData:
  UNIFI_API_KEY: <redacted>

With the gateway-httproute source, every HTTPRoute attached to my Cilium Gateway gets its hostname published to UniFi, pointing at the gateway’s LoadBalancer IP, the very one BGP just made reachable. Deploying a new service is now a Deployment, a Service, and an HTTPRoute, and DNS sorts itself out.

Records that don’t live in the cluster

Not everything I want in DNS runs in K8s. Proxmox hosts, Home Assistant, the NFS box. For those, the crd source lets me declare records as a DNSEndpoint resource, so even my “static” DNS entries are YAML in git instead of clicks in a UI:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: custom-records
  namespace: external-dns
spec:
  endpoints:
    - dnsName: homeassistant.mydomain
      recordType: A
      recordTTL: 300
      targets:
        - 10.42.1.100
    - dnsName: proxmox.mydomain
      recordType: A
      recordTTL: 300
      targets:
        - 10.42.0.10

The DNSEndpoint CRD itself is not installed by the chart, you grab it from the external-dns repository and apply it yourself. In my case Argo CD handles that too. The Application is a multi-source one: the first source is the Helm chart with the values shown above, the second is a path in my git repository holding the CRD, the DNSEndpoint resources, and the secret:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-dns
  namespace: argocd
spec:
  project: infrastructure
  sources:
    - chart: external-dns
      repoURL: https://kubernetes-sigs.github.io/external-dns/
      targetRevision: 1.21.1
      helm:
        releaseName: external-dns
        values: |
          # values from above
    - repoURL: git@git.mydomain:ivan/k8s-apps.git
      targetRevision: main
      path: external-dns
  destination:
    server: https://kubernetes.default.svc
    namespace: external-dns
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Argo CD applies everything in that path as plain manifests, CRD included, so the chart and the resources it needs land together in one sync.

One note on policy: sync: it means external-dns will also delete records, which sounds scary for a DNS server that holds the whole household together. The ownership TXT records, tagged with the txtOwnerId, make sure the controller only manages records it created itself. My manually created records from the before times are safe until I get around to converting them.

Closing thoughts

The whole thing is around a hundred lines of YAML plus one FRR config, and I have not opened the DNS page in the UniFi UI since. The router learns routes from the cluster and the cluster writes names into the router, which feels like the two of them finally agreed to talk to each other without me playing interpreter .