Backstory
A few weeks ago (not today, and by the time I published this post it’s a few months ago), I fucked up: During irregular disk cleanup, I accidentally removed the config folder!
Fortunately, I “backed up” the config with git, didn’t I? 😌
Well yes, but actually no! 🥲 I did commit, but I didn’t push. The latest pushed config had been 9 months before this incident.
On top of that, many programs generate unreadable configs, which I prefer not
to put into version control, or put sensitive information or large data into
.config, which shouldn’t be pushed into a remote repository on a server I
don’t own. For example, nheko puts authentication info there, and after this
incident I lost the session with several messages—well, matrix’s irregularly
regular decryption failure could be expanded into a post on its own, so let’s
not digress.
And then there are more things to back up than just configs, say, my photos and music. I mirror them between my devices with syncthing, providing redundancy, but syncthing is not backup, and neither is RAID
I should back them up, or I might one day be doomed to repeat the mistake.
Preparation
Despite the bitterness from the partial data loss, I kinda didn’t have the spoon to do what I must, so I’ve been delaying it hitherto.
Hardware
One of the reason I was cleaning up the disk in the first place was that it’s start to be filled up, so I bought a large hard drive to store more data and backup. Doesn’t seem too safe, I know: if the hard drive gets lost or broken, I’ll lose both the main data and the backup, but with a limited budget and space, that’s good enough for now. I am planning for some off-site backup.
I got a Seagate One Touch 2 TB1 HDD. It seems to come with builtin backup software, which doesn’t support Linux of course, and I likely wouldn’t use it even if it did. A USB hub is also needed, as my laptop has a limited number of USB ports.
Formatting
Coming with the hard drive are several files, such as an .exe program to
initialize the backup, I suppose. I don’t need these, and the disk can be
formatted right away after I do full-disk encryption on that.
cryptsetup luksFormat /dev/sdb
cryptsetup open --type luks /dev/sdb extern
No partitioning is needed: while I do seek vegan alternative, I intended to use btrfs as the file system, which can create subvolumes, which can act as separate mounted partitions. I would just reformat the existing partition to btrfs.
mkfs.btrfs -L margarine /dev/mapper/extern
I intended to have three subvolumes, one for big data (not Big Data™) such as movies or music, one for more personal data and another for backup:
mkdir /data
mount /dev/mapper/extern /data
btrfs subvolume create /data/hoard
btrfs subvolume create /data/perso
btrfs subvolume create /data/backup
ACCIDENT!
OK, I’ll admit I don’t know what the heck went on, but the main user, xarvos,
and root were apparently deleted, but not actually. I cannot log in to either
account, but they were still listed in the /etc/passwd file. Huh? I
cancelled a rebuild, which I have done a dozen times before and nothing went
wrong so it should be fine this time, right? Right? (Looking at
the laptop Padmely)
Per later investigation, it looks like for some unknown reason, the NixOS configurations were removed mistakenly, and the failure was triggered by a rebuild.
Rescue
Well, it’d be a lie to say I didn’t panic. But this is not something not having happened before (translation for people confused by double negatives: similar thing happened before).
The easiest solution is to reinstall the thing (though it will leave me wondering why this happenned). At the time of my NixOS installation, I didn’t make use of the service configurations in nix, only declaring those packages and invoke them manually. Additionally, like cnx, I also didn’t think of full-disk encryption during installation. So, this is an opportunity to rectify those wrongs
Backing up
Thanks to the available external disk as well as a ready bootable disk (you should always have those, for occasions like these), I can simply back up my whole home directory. I don’t do stuff elsewhere, so no needs to back those up.
mkdir /mnt/{int,ext}
mount /dev/sda1 /mnt/int
cryptsetup open --type luks /dev/sdb extern
mount /dev/mapper/extern /mnt/ext
rsync -a /mnt/int/home/xarvos /mnt/ext/backup/manual-$(date)-home-xarvos
This also mean I can do the rescue while documenting this process at the same time 😃.
The bootable USB drive is a NixOS installer, but it’s not the latest version, so I burn a newer installer to a second one (so handy), after backing up the second one’s content to the disk using similar process.
Reinstallation
OK, it’s time to remount. I can install right now with nixos-install --root=/mnt/int, but I want to include the external disk in the same root, so
that their hardware configuration can be generated together. Though, before
remounting, I should encrypt the main partition.
Previously, I only have two partition—the main one and the swap, but now I
would need three, because the /boot can’t be encrypted
parted /dev/sda
(parted) mklabel msdos
(parted) mkpart primary 128MB -8GB
(parted) mkpart primary linux-swap -8GB 100%
(parted) mkpart primary 1MB 128MB
(parted) set 3 boot on
mkswap /dev/sda2
swapon /dev/sda2
mkfs.ext4 -L boot /dev/sda3
cryptsetup luksFormat /dev/sda1
cryptsetup open --type luks /dev/sda1 intern
mkfs.btrfs -L pain /dev/mapper/intern
(pain as in French for bread, because of course we spread margarine on bread :wink:)
mount /dev/mapper/intern /mnt
mkdir /mnt/data
mount /dev/mapper/extern /mnt/data
Before generating the config, I’d like to make some more subvolumes. I didn’t intend to run with my root on tmpfs like cnx, but I believe it’d be better to have different partitions to be backupped (with snapshots) separately.
btrfs subvolume create /mnt/etc
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/var
After that, I copied the configuration to /etc and generated the config.
mkdir /mnt/etc/nixos
cp -r /mnt/data/backup/.../{services,configuration.nix} /mnt/etc/nixos
nixos-generate-config --root /mnt
(Because nixos-generate-config does not overwrite configuration.nix file if
it already exists, I can copy the configs there first.)
While in an urgence (there was one day and a half before the holiday period
ended and I needed the machine back for work), I still spent the time to rewrite
some services, namely syncthing, transmission, and mpd. For example, I set mpd
home to /data/hoard
services.mpd = {
enable = true;
user = "xarvos";
dataDir = "/data/hoard/music";
startWhenNeeded = true;
};
Finally, I ran nixos-install to finish the setup, and sync’ed the backup back home.
Setting up
(About two months later)
After rebuilding the system, I kinda ran out of energy for doing the backup (again), so I postponed the setup to the week after, but then the next week something happened, which delayed it further and further, and since the momentum died, it took quite a while for me to pick up this again.
My initial plan was to use snapper, as I had good experience with it back when I was using openSUSE.
Snapper (HTTP without TLS link, no idea why) is a tool by openSUSE project that helps managing file system snapshots, which was intended for btrfs, but also supports several other file systems. See also its source, the tutorial on opensuse.org, and ArchLinux wiki article for more information. My main use of snapper was pre-post snapshots, which was integrated into zypper, but I have no use of that on NixOS, since I already manage system changes in git, and NixOS also create different boot options on rebuild as well. Timeline snapshot is also great, but it seems a bit overkill for me. Additionally, snapper doesn’t seem to provide a way to send snapshots to the external drive, so I’d have to set that up, and it’d probably be harder to get the names correctly.
So, I’d just copy cnx here, which is less thinking
btrfs subvolume create /data/backup/home
today=$(date --iso-8601)
btrfs subvolume snapshot -r /home /home/$today
sync
btrfs send /home/$today | btrfs receive /data/backup/home
sync
Now, I have to setup a cron job to automate backup, as well as deletion. I modified the script a little so running it twice does not accidentally delete last snapshot.
previous=$(find /home/ -maxdepth 1 -regex '.*20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]')
today=$(date --iso-8601)
if [[ "$previous" == "/home/$today" ]]; then
echo "Today is already backed up"
else
echo "Running backup"
btrfs subvolume snapshot -r /home /home/$today
btrfs send -p $previous /home/$today | btrfs receive /data/backup/home
btrfs subvolume delete $previous
sync
fi
Then, I set up the fcron service. Following the example in
fcrontab manual, I can use %hours keyword so that this runs once
in the day from 9h to 16h:
services.fcron = {
enable = true;
systab = "%hours 0 9-16 * * * /path/to/backup.sh"
};
I think this is pretty much it. I will wait till tomorrow to see if the cron will run properly.