{"id":12,"date":"2026-07-31T13:48:41","date_gmt":"2026-07-31T20:48:41","guid":{"rendered":"https:\/\/pointyhair.com\/?p=12"},"modified":"2026-07-31T15:26:11","modified_gmt":"2026-07-31T22:26:11","slug":"using-lvm2-for-a-resilient-drive-array","status":"publish","type":"post","link":"https:\/\/pointyhair.com\/index.php\/2026\/07\/31\/using-lvm2-for-a-resilient-drive-array\/","title":{"rendered":"Using LVM2 for a resilient scalable drive array"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">63 terabytes of storage built from 8 USB drives in RAID 1 configuration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A bit of background:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>I started a (multi-domain) web service ten years ago. I wanted some extra storage as backups.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fairly quickly I was running out of space, as well as backups of backups &#8212; so I had to add storage to my USB drive. Now I wanted to serve the pages of an NVME (originally M2.SATA)  drive, and have the backups on a slower (cheaper medium). I started to grow the number of USB drives.<\/li>\n\n\n\n<li>The number of USB drives became unwieldy because if the server (Linux) reboots, the drives come in an undetermined order. Sorting out the drives between sdb, sdc&#8230;sdi is not fun.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">To start with&#8230; this is a completely safe, benign operation &#8212; it just makes LVM2 available to your machine:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install lvm2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Enter LVM2, which I had used in the past.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo pvs\u00a0\nPV \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0VG \u00a0\u00a0Fmt \u00a0Attr PSize \u00a0PFree \u00a0\u00a0\u00a0\n\u00a0\/dev\/sdb \u00a0\u00a0big2 lvm2 a-- \u00a0&lt;4.55t \u00a0\u00a0&lt;1.55t\n\u00a0\/dev\/sdc \u00a0\u00a0big \u00a0lvm2 a-- \u00a0&lt;7.28t &lt;284.04g\n\u00a0\/dev\/sdd \u00a0\u00a0big \u00a0lvm2 a-- \u00a0&lt;7.28t \u00a0\u00a075.07g\n\u00a0\/dev\/sde \u00a0\u00a0big \u00a0lvm2 a-- \u00a0&lt;7.28t \u00a0\u00a0\u00a0\u00a0\u00a0\u00a00 \u00a0\n\u00a0\/dev\/sdf \u00a0\u00a0big2 lvm2 a-- \u00a0&lt;3.64t \u00a0\u00a0&lt;3.64t\n\u00a0\/dev\/sdg \u00a0\u00a0big \u00a0lvm2 a-- \u00a0&lt;7.28t &lt;284.04g\n\u00a0\/dev\/sdh \u00a0\u00a0big \u00a0lvm2 a-- \u00a012.73t \u00a0\u00a010.73t\n\u00a0\/dev\/sdi \u00a0\u00a0big2 lvm2 a-- \u00a014.55t \u00a0818.96g<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">I took all the USB drives, ran PVcreate (Physical Volume create) on each drive, adding them to a Volume Groug (VG). I realized that the drives could be categorized as Slow and Fast, depending on their seek times (and whether they &#8220;fell asleep&#8221; between access). So I added the PVs into <strong>big<\/strong> and <strong>big2<\/strong> groups [using VGcreate](not the best naming, should have used Fast and Slow).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo vgs\u00a0\nVG \u00a0\u00a0#PV #LV #SN Attr \u00a0\u00a0VSize \u00a0\u00a0VFree \u00a0\n\u00a0big \u00a0\u00a0\u00a05 \u00a028 \u00a0\u00a00 wz--n- \u00a041.84t 11.36t\n\u00a0big2 \u00a0\u00a03 \u00a0\u00a09 \u00a0\u00a00 wz--n- &lt;22.74t &lt;5.99t\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Finally I started creating Logical Volumes (LVcreate) for each purpose always creating LVcreate on VG1 and VG2 same size, and creating an rsync between the LVs (backup of a backup &#8212; effectively bring the whole array into a RAID1 configuration). Now, I could have used LVM2s native tools to do this as well, but I wanted more control (and I had been burned by LVM1 in the pre-historic 10year+ past).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Nowadays, I have 38 Logical volumes (spread across 9 PVs), that I occasionally expand or shink depending on size needed, usually about a terabyte at a time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My LVS map is too long (and private to show).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s the killer!<br>When the California heatwave occured, I lost the machine (srv3) completely. and I built a new server (srv4), I just took all the USB hubs from srv3, plugged into srv4, and ran PVSCAN. LVM2 rebuilt the whole raid array, with configurations (PVS, and VGs). All I needed to do was copy the \/etc\/fstab from a backup of srv3&#8230; and we&#8217;re back in business!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The USBs, the drive IDs, the HUBs don&#8217;t matter, everything fell back in place correctly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The only thing I wish is that I hoped I hade done better job in taking mysqldumps of the databases instead of doing rsyncs, thus not needing to rebuild the sites. But the sites that were HTML came back immediately.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Emails on user accounts were a bit trickier, Folder by folder reconstruction became a possibility, though some tricks were involved (sendmail, dovecot &#8212; I will probably write another article). I will likely add more details to this if interested.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>63 terabytes of storage built from 8 USB drives in RAID 1 configuration. A bit of background: To start with&#8230; this is a completely safe, benign operation &#8212; it just makes LVM2 available to your machine: Enter LVM2, which I had used in the past. I took all the USB drives, ran PVcreate (Physical Volume [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-12","post","type-post","status-publish","format-standard","hentry","category-linux-system-configuration"],"_links":{"self":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":7,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12\/revisions"}],"predecessor-version":[{"id":30,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/posts\/12\/revisions\/30"}],"wp:attachment":[{"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/media?parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/categories?post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pointyhair.com\/index.php\/wp-json\/wp\/v2\/tags?post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}