63 terabytes of storage built from 8 USB drives in “RAID 1” configuration — I.e. you don’t lose data even if one drive goes dead.<\/p>\n\n\n\n
A bit of background:<\/p>\n\n\n\n
I started a (multi-domain) web service ten years ago. I wanted some extra storage as backups.<\/p>\n\n\n\n
To start with… this is a completely safe, benign operation — it just makes LVM2 available to your machine:<\/p>\n\n\n\n
sudo apt install lvm2<\/code><\/pre>\n\n\n\nEnter LVM2, which I had used in the past.<\/p>\n\n\n\n
sudo pvs \nPV VG Fmt Attr PSize PFree \n \/dev\/sdb big2 lvm2 a-- <4.55t <1.55t\n \/dev\/sdc big lvm2 a-- <7.28t <284.04g\n \/dev\/sdd big lvm2 a-- <7.28t 75.07g\n \/dev\/sde big lvm2 a-- <7.28t 0 \n \/dev\/sdf big2 lvm2 a-- <3.64t <3.64t\n \/dev\/sdg big lvm2 a-- <7.28t <284.04g\n \/dev\/sdh big lvm2 a-- 12.73t 10.73t\n \/dev\/sdi big2 lvm2 a-- 14.55t 818.96g<\/code><\/pre>\n\n\n\nI took all the USB drives, ran PVcreate (Physical Volume create) on each drive, adding them to a Volume Groug (VG). I realized that the drives could be categorized as Slow and Fast, depending on their seek times (and whether they “fell asleep” between access). So I added the PVs into big<\/strong> and big2<\/strong> groups [using VGcreate](not the best naming, should have used Fast and Slow).<\/p>\n\n\n\nsudo vgs \nVG #PV #LV #SN Attr VSize VFree \n big 5 28 0 wz--n- 41.84t 11.36t\n big2 3 9 0 wz--n- <22.74t <5.99t\n<\/code><\/pre>\n\n\n\nFinally I started creating Logical Volumes (LVcreate) for each purpose always creating LVcreate on VG1 and VG2 same size, and creating an rsync between the LVs (backup of a backup — effectively bring the whole array into a RAID1 configuration). Now, I could have used LVM2s native tools to do this as well, but I wanted more control (and I had been burned by LVM1 in the pre-historic 10year+ past).<\/p>\n\n\n\n
Nowadays, I have 38 Logical volumes (spread across 9 PVs), that I occasionally expand or shink depending on size needed, usually about a terabyte at a time.<\/p>\n\n\n\n
My LVS map is too long (and private to show).<\/p>\n\n\n\n
Here’s the killer!
When the California heatwave occured, I lost the machine (srv3) completely. and I built a new server (srv4), I just took all the USB hubs from srv3, plugged into srv4, and ran PVSCAN. LVM2 rebuilt the whole raid array, with configurations (PVS, and VGs). All I needed to do was copy the \/etc\/fstab from a backup of srv3… and we’re back in business!<\/p>\n\n\n\n
The USBs, the drive IDs, the HUBs don’t matter, everything fell back in place correctly.<\/p>\n\n\n\n
The only thing I wish is that I hoped I hade done better job in taking mysqldumps of the databases instead of doing rsyncs, thus not needing to rebuild the sites. But the sites that were HTML came back immediately.<\/p>\n\n\n\n
I had a bad USB hub, causing some PVs (Physical Volumes) to go off-line, and since I have many of the Logical Volumes shared\/mounted via NFS that became a problem. Here is a script that 1) takes NFS off-line, 2) checks all the mount points, 3) re-mounts them, 4) restarts NFS. Easy to run, All shares restored, all volumes re-mounted.<\/p>\n\n\n\n
cat \/root\/reseatusb.sh \n#\/bin\/bash\n\nsudo \/etc\/init.d\/nfs-kernel-server stop\nDF1=$(df -h|grep \/dev\/mapper|awk '{print $1}')\nDF6=$(df -h|grep \/dev\/mapper|awk '{print $6}')\necho $DF6|xargs umount\necho $DF1|xargs lvchange -an \necho $DF1|xargs lvchange -ay \nmount -a\nsudo \/etc\/init.d\/nfs-kernel-server start\n<\/code><\/pre>\n\n\n\nOf course, once I realized it was because the bad hub, I replaced with another USB3 hub, and I’ve never needed the script anymore.<\/p>\n\n\n\n
Emails on user accounts were a bit trickier, Folder by folder reconstruction became a possibility, though some tricks were involved (sendmail, dovecot — I will probably write another article). I will likely add more details to this if interested.<\/p>\n","protected":false},"excerpt":{"rendered":"
63 terabytes of storage built from 8 USB drives in “RAID 1” configuration — I.e. you don’t lose data even if one drive goes dead. A bit of background: I…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"slim_seo":{"title":"Using LVM2 for a resilient scalable drive array, with script to re-initialize array - Pointyhair","description":"63 terabytes of storage built from 8 USB drives in \"RAID 1\" configuration -- I.e. you don't lose data even if one drive goes dead. A bit of background: I starte"},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-12","post","type-post","status-publish","format-standard","hentry","category-linux-system-configuration"],"_links":{"self":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":10,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12\/revisions"}],"predecessor-version":[{"id":64,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12\/revisions\/64"}],"wp:attachment":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/media?parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/categories?post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/tags?post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}