My main rig is running Ubuntu 18.04. To be more precise I opted out at install time to use Ubuntu Mate 18.04 but later on installed AwesomeWM and use that instead now. But without digressing much, I decided it was time to move my root (/) to ZFS. Why? - Because it is awesome!
ZFS is my favorite FS of choice for some time now. I don’t use it everywhere (am trying to be smart about it) but I prefer to do whenever I have the chance.
This main PC has 2 NVMe drives and root was installed on only one of those, second one was basically unused for the whole time (experimentation and stuff) so I had the wiggle room to do the migration.
- Create ZPOOL on that spare NVMe
- Copy system files to that new ZPOOL
- Chroot and make required changes
- Install bootloader
Note that my system is using UEFI so steps regarding bootloader preparation differ from MBR boot type. Just follow the official Wiki and you’ll be good.
Article from ZFSonLinux wiki page on Github was the main guideline (basically I C/P everything from there).
Destroy everything on the drive and create 2 partitions (EFI and partition which pool will be created
sgdisk --zap-all /dev/disk/by-id/nvme-Force_MP500_17047932000122530589 # Create EFI Partition sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/nvme-Force_MP500_17047932000122530589 # Create zpool partition sgdisk -n1:0:0 -t1:BF01 /dev/disk/by-id/nvme-Force_MP500_17047932000122530589
Now that the disk is prepared it is time to create ZPOOL and required filesystems on it
# Create pool zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD -O xattr=sa -O mountpoint=/ -R /mnt rpool nvme-Force_MP500_17047932000122530589-part1 # Create filesystem dataset to act as a container (like on FreeBSD) zfs create -o canmount=off -o mountpoint=none rpool/ROOT # Root filesystem zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu # Mount filesystem (to /mnt ^^) zfs mount rpool/ROOT/ubuntu
Once the main part is done you can create datasets. I’ve omitted /home as I already use other ZFS pool as my /home
# Create root zfs create -o mountpoint=/root rpool/root zfs create -o canmount=off -o setuid=off -o exec=off rpool/var zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o acltype=posixacl -o xattr=sa rpool/var/log zfs create rpool/var/spool zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/var/tmp zfs create rpool/srv zfs create rpool/var/games zfs create rpool/var/mail zfs create -o com.sun:auto-snapshot=false -o mountpoint=/var/lib/docker rpool/var/docker zfs inherit exec rpool/var zfs create -o com.sun:auto-snapshot=false -o setuid=off rpool/tmp chmod 1777 /mnt/tmp
Copy system files
In this step I basically just used rsync, but I first had to mount my old systems to some new path, so I wouldn’t copy over /dev, /proc and other files as that would cause issues
# Create temp folder mkdir /oldroot # Mount current root to it (You CAN do this while system is live) mount /dev/nvme1n1p3 /oldroot # Mount current boot to it mount /dev/nvme1n1p2 /oldroot/boot
With that out of the way we can start copying the files:
rsync -arAXHvW /oldroot/ /mnt/
Chroot and make required changes
First we need to mount standard directories
mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys chroot /mnt /bin/bash --login
Once I was there I’ve found that system wouldn’t resolve URLs (no systemd-resolved running) so I’ve just added:
To the /etc/resolv.conf file
From there we had to install required packages:
apt install --yes --no-install-recommends linux-image-generic apt install --yes zfs-initramfs
Without proper initramfs system would refuse to boot.
Now I had to create FS for EFI partition
apt install dosfstools mkdosfs -F 32 -n EFI /dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3 echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3) /boot/efi vfat noatime,nofail,x-systemd.device-timeout=1 0 1 > /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64
Note the > /etc/fstab part. I’ve nuked it intentionally, you may not want to do that, so use brain and review what you need and what can/needs to be removed.
Datasets /var/log, /var/tmp and /tmp, if you leave them to be mounted by zfs import cache, will cause issues with race conditions between order of filesystem mounting (done by zfs import service) and daemons which expect some filesystems to already be available, so the solution is to switch them to legacy mountpoint and handle them in fstab
zfs set mountpoint=legacy rpool/var/log zfs set mountpoint=legacy rpool/var/tmp zfs set mountpoint=legacy rpool/tmp cat >> /etc/fstab << EOF rpool/var/log /var/log zfs noatime,nodev,noexec,nosuid 0 0 rpool/var/tmp /var/tmp zfs noatime,nodev,nosuid 0 0 rpool/tmp /tmp zfs noatime,nodev,nosuid 0 0 EOF
Ensure we mount tmpfs to /tmp
cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount
Before proceeding you can verify from chroot that root filesystem is recognized by the grub
grub-probe / # should output "zfs"
Update initramfs files, grub and install grub
update-initramfs -u -k all update-grub grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck --no-floppy
Verify that module for ZFS is installed
At this point I’ve found that unmounting things and rebooting gracefully didn’t work, you may try to exit chroot, umount all filesystems from /mnt and reboot, good luck! When my reboot stopped I just rebooted the system by pressing the reset button.
Add second drive to the rpool
Once the system booted up I’ve nuked the old drive (holding old root) and attached it to the rpool so I have redundancy (mirror).
To achieve that I’ve used:
sgdisk --zap-all /dev/disk/by-id/nvme-Force_MP500_17047932000122530587 # Create EFI Partition sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587 # Create zpool partition sgdisk -n1:0:0 -t1:BF01 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587 # Attach drive to the existing rpool zpool attach rpool /dev/disk/by-id/nvme-Force_MP500_17047932000122530589 /dev/disk/by-id/nvme-Force_MP500_17047932000122530587
Note that you have to specify current zpool drive first so it will attach it to it as a mirror, and not do the striping (raid 0) except that is what you’re aiming for.
And to ensure we can still boot if one drive fails we need to have efi on this drive as well so just dd partition from the first drive to the second one:
umount /boot/efi dd if=/dev/disk/by-id/nvme-Force_MP500_17047932000122530589-part3 of=/dev/disk/by-id/nvme-Force_MP500_17047932000122530587-part3 efibootmgr -c -g -d /dev/disk/by-id/nvme-Force_MP500_17047932000122530587 -p 3 -L "ubuntu-2" -l '\EFI\Ubuntu\grubx64.efi' mount /boot/efi