Introduction

For the past few months, I’ve been on a journey of moving all of my self-hosted stuff into kubernetes in order to learn it better. During this time I have also deployed a bunch of new apps in order to replace some of the hosted solutions I used prior to this in hope to save some money on the subscriptions, and with time, to reinvest in my home lab further.

One of this apps has been excellent Audiobookshelf, to replace both PocketCasts for podcasts and Audible for audio books. I have been amazed by the app during the past few months I have been using it.

Initially when I decided to deploy Audiobookshelf on my K8s cluster, I pieced together a StatefulSet definition which created one instance of the app (1 pod) and PersistentVolumeClaim for each of the volumeMounts (config, metadata, podcasts, audiobooks).

When I started I was conservative with the space allocated to the volumes via volumeClaimTemplates definition, which meant I quickly ran out of space while the actual usage hit the service (podcasts started downloading, I started uploading audio books etc.). I then quickly realized that I can’t simply change the StatefulSet manifest and apply it to resize the volumes, instead, I had to reside to the manual volume resize which was no big deal.

As I deploy all of the apps using ArgoCD now (which is beyond the scope of this blog post), this meant that I either had to:

have ArgoCD not being able to sync the state of the StatefulSet if I changed the manifest in my repository, and thus not be able to manage deployments
have value that doesn’t match the reality in the volumeClaimTemplates section in git, which triggers my CDO (OCD in the alphabetical order) and also means I’m managing volume size from that point manually and directly on the cluster

I’m sure StatefulSet has its use-cases (like for managing database instances for example), but I’m now thinking this app wasn’t really a good fit for it, so I decided to transition to the Deployment

Easier said than done (not quite)

As this is not a mission critical operation, I had the privilege of scaling app to 0, and make necessary steps. Initial plan was quite simple:

write definition for each of the PV and PVC resources and commit to git
comment out StatefulSet definition
write Deployment definition
let ArgoCD sort things out by syncing it to the cluster

I haven’t given this too much thought as I quickly got some other ideas in my mind, so perhaps such scenario would not even work in the first place (I would probably realize it too late and have to restore things from the backup).

As I’m (still) provisioning and managing persistent volumes using Longhorn, and as podcasts and audio books had 60+GB worth of data in them, and as that was a cause of frequent problems with my kubernetes storage (one example: node crashes for misc reasons, longhorn starts re-sync in order to ensure number of desired copies, grinds other nodes to halt as that is IO intensive with that amount of data, node comes back up, longhorn triggers re-sync again to ensure no 2 copies are on the same node etc. Perhaps I’ll go into detail about this part of the setup sometime in the future(tm), but I’m not currently very happy with Longhorn as it stands) I came up with a brilliant idea.

Anyhow, my brilliant idea (almost as brilliant as I, myself, am) was to provision those large volumes using NFS (I have nfs-csi configured already in the cluster), and keep config and metadata on longhorn cluster (it should in theory be more performant since it uses local storage).

This required some variations in the initial plan, so the process went somewhat like this:

create PV (NFS provisioned)
- podcasts
- audiobooks
create PVC for each of the volumes
- podcasts -> NFS volume
- audiobooks -> NFS volume
- config -> longhorn
- metadata -> longhorn

Longhorn will take care of creating new volumes for the PVC. I also used different (simpler) names to make everything clean and consistent, which also resulted with added benefit that I could easily revert back to the old setup if something went wrong horribly wrong.

With all pieces in place, it was time to migrate the data. In order to avoid shenanigans with mounting volumes to nodes manually, syncing things etc. I found a small utility called pv-migrate that did that for me. It basically created a new pod which mounted both the old and the new volume, and synced the data to the new volume using rsync. I used this for each of the 4 volumes.

Once the data migration process was complete, I applied the Deployment definition and application started up with 1 replica. I then verified if everything worked like and after confirming so, I deleted the StatefulSet.

Audiobookshelf configuration

For anyone interested, here’s my full up-to-the-time-of-publishing-this-article service definition (perhaps I’ll open up git repository and link it here at one point in the future):

# namespace.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: audiobookshelf

# service.yaml
kind: Service
apiVersion: v1
metadata:
  name: audiobookshelf
  namespace: audiobookshelf
spec:
  selector:
    app: audiobookshelf
  ports:
  - protocol: TCP
    port: 80
    name: http

# networking.yaml
# generated with https://editor.networkpolicy.io/
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: audiobookshelf-network-policy
  namespace: audiobookshelf
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: haproxy-controller
          podSelector:
            matchLabels:
              app.kubernetes.io/name: kubernetes-ingress
      ports:
        - port: 80
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP


# ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: audiobookshelf
  namespace: audiobookshelf
spec:
  ingressClassName: haproxy
  rules:
  - host: audiobookshelf.MYINTERNALDNS
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: audiobookshelf
            port:
              name: http

# volumes.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: audiobookshelf-podcasts
  annotations:
    pv.kubernetes.io/provisioned-by: nfs.csi.k8s.io
spec:
  capacity:
    storage: 2Ti
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - nfsvers=4.1
  csi:
    driver: nfs.csi.k8s.io
    volumeHandle: st00rage.MYINTERNALDNS.#volume1#audiobookshelf-podcasts
    volumeAttributes:
      server: st00rage.MYINTERNALDNS.
      share: /volume1/audiobookshelf-podcasts
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: audiobookshelf-audiobooks
  annotations:
    pv.kubernetes.io/provisioned-by: nfs.csi.k8s.io
spec:
  capacity:
    storage: 2Ti
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - nfsvers=4.1
  csi:
    driver: nfs.csi.k8s.io
    volumeHandle: st00rage.MYINTERNALDNS.#volume1#audiobookshelf-audiobooks
    volumeAttributes:
      server: st00rage.MYINTERNALDNS.
      share: /volume1/audiobookshelf-audiobooks
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: audiobookshelf-podcasts
  namespace: audiobookshelf
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  volumeName: audiobookshelf-podcasts
  storageClassName: ""
  resources:
    requests:
      storage: 2Ti
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: audiobookshelf-audiobooks
  namespace: audiobookshelf
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  volumeName: audiobookshelf-audiobooks
  storageClassName: ""
  resources:
    requests:
      storage: 2Ti
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: audiobookshelf-config
  namespace: audiobookshelf
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 128Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: audiobookshelf-metadata
  namespace: audiobookshelf
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 512Mi

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: audiobookshelf
  namespace: audiobookshelf
  labels:
    app: audiobookshelf
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: audiobookshelf
  template:
    metadata:
      labels:
        app: audiobookshelf
    spec:
      containers:
        - name: audiobookshelf
          image: ghcr.io/advplyr/audiobookshelf:2.9.0
          volumeMounts:
            - name: config
              mountPath: /config
            - name: metadata
              mountPath: /metadata
            - name: podcasts
              mountPath: /podcasts
            - name: audiobooks
              mountPath: /audiobooks
      volumes:
        - name: config
          persistentVolumeClaim:
            claimName: audiobookshelf-config
        - name: metadata
          persistentVolumeClaim:
            claimName: audiobookshelf-metadata
        - name: audiobooks
          persistentVolumeClaim:
            claimName: audiobookshelf-audiobooks
        - name: podcasts
          persistentVolumeClaim:
            claimName: audiobookshelf-podcasts

I’m aware that there are plenty of things that can be improved, but so far, this works fine for me. If you see some mistakes, or have some advice how to improve it though, don’t hesitate to reach out, I’m always glad to hear that someone reads this stream of semi-filtered thoughts and unnecessarily long articles on my blog :-)

Outro

During my learning process, I was bound to make some mistakes, and I will probably make a bunch more, but hey, that’s life.

I know I have been slow in posting certain stuff on this blog, but as you all know, life happens, so I’ll do that whenever I have some extra time and energy to write about certain topics. If you have some suggestions or something you’d like to read about next in my setup, feel free to reach out via email or something :-)

Introduction#

Easier said than done (not quite)#

Audiobookshelf configuration#

Outro#

Introduction

Easier said than done (not quite)

Audiobookshelf configuration

Outro