63 terabytes of storage built from 8 USB drives in RAID 1 configuration.
A bit of background:
- I started a (multi-domain) web service ten years ago. I wanted some extra storage as backups.
- Fairly quickly I was running out of space, as well as backups of backups — so I had to add storage to my USB drive. Now I wanted to serve the pages of an NVME (originally M2.SATA) drive, and have the backups on a slower (cheaper medium). I started to grow the number of USB drives.
- The number of USB drives became unwieldy because if the server (Linux) reboots, the drives come in an undetermined order. Sorting out the drives between sdb, sdc…sdi is not fun.
To start with… this is a completely safe, benign operation — it just makes LVM2 available to your machine:
sudo apt install lvm2Enter LVM2, which I had used in the past.
sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sdb big2 lvm2 a-- <4.55t <1.55t
/dev/sdc big lvm2 a-- <7.28t <284.04g
/dev/sdd big lvm2 a-- <7.28t 75.07g
/dev/sde big lvm2 a-- <7.28t 0
/dev/sdf big2 lvm2 a-- <3.64t <3.64t
/dev/sdg big lvm2 a-- <7.28t <284.04g
/dev/sdh big lvm2 a-- 12.73t 10.73t
/dev/sdi big2 lvm2 a-- 14.55t 818.96gI took all the USB drives, ran PVcreate (Physical Volume create) on each drive, adding them to a Volume Groug (VG). I realized that the drives could be categorized as Slow and Fast, depending on their seek times (and whether they “fell asleep” between access). So I added the PVs into big and big2 groups [using VGcreate](not the best naming, should have used Fast and Slow).
sudo vgs
VG #PV #LV #SN Attr VSize VFree
big 5 28 0 wz--n- 41.84t 11.36t
big2 3 9 0 wz--n- <22.74t <5.99t
Finally I started creating Logical Volumes (LVcreate) for each purpose always creating LVcreate on VG1 and VG2 same size, and creating an rsync between the LVs (backup of a backup — effectively bring the whole array into a RAID1 configuration). Now, I could have used LVM2s native tools to do this as well, but I wanted more control (and I had been burned by LVM1 in the pre-historic 10year+ past).
Nowadays, I have 38 Logical volumes (spread across 9 PVs), that I occasionally expand or shink depending on size needed, usually about a terabyte at a time.
My LVS map is too long (and private to show).
Here’s the killer!
When the California heatwave occured, I lost the machine (srv3) completely. and I built a new server (srv4), I just took all the USB hubs from srv3, plugged into srv4, and ran PVSCAN. LVM2 rebuilt the whole raid array, with configurations (PVS, and VGs). All I needed to do was copy the /etc/fstab from a backup of srv3… and we’re back in business!
The USBs, the drive IDs, the HUBs don’t matter, everything fell back in place correctly.
The only thing I wish is that I hoped I hade done better job in taking mysqldumps of the databases instead of doing rsyncs, thus not needing to rebuild the sites. But the sites that were HTML came back immediately.
Emails on user accounts were a bit trickier, Folder by folder reconstruction became a possibility, though some tricks were involved (sendmail, dovecot — I will probably write another article). I will likely add more details to this if interested.