Using LVM/LVM2 for a resilient scalable drive array, with script to re-initialize array

Posted on by Taneli

63 terabytes of storage built from 8 USB drives in “RAID 1” configuration — I.e. you don’t lose data even if one drive goes dead.

No limitations by the size of one disk/volume that you add, instead all disks are combined together in volumes (and you have multiple columes) that you define, the you slice parts of the volumegroup as you desire. You and shrink and expand the groups.

A bit of background:

I started a (multi-domain) web service ten years ago. I wanted some extra storage as backups.

  • Fairly quickly I was running out of space, as well as backups of backups — so I had to add storage to my USB drive. Now I wanted to serve the pages out of an NVME (originally M2.SATA) drive, and have the backups on a slower (cheaper medium). I started to grow the number of USB drives.
  • The number of USB drives became unwieldy because if the server (Linux) reboots, the drives come in an undetermined order. Sorting out the drives between sdb, sdc…sdi is not fun.
  • Here is another good youtube, getting basic LVM working: https://www.youtube.com/watch?v=Ap-S2faYjbA

To start with… this is a completely safe, benign operation — it just makes LVM2 available to your machine:

sudo apt install lvm2

Enter LVM2, which I had used in the past.

sudo pvs 
PV         VG   Fmt  Attr PSize  PFree    
 /dev/sdb   big2 lvm2 a--  <4.55t   <1.55t
 /dev/sdc   big  lvm2 a--  <7.28t <284.04g
 /dev/sdd   big  lvm2 a--  <7.28t   75.07g
 /dev/sde   big  lvm2 a--  <7.28t       0  
 /dev/sdf   big2 lvm2 a--  <3.64t   <3.64t
 /dev/sdg   big  lvm2 a--  <7.28t <284.04g
 /dev/sdh   big  lvm2 a--  12.73t   10.73t
 /dev/sdi   big2 lvm2 a--  14.55t  818.96g

I took all the USB drives, ran PVcreate (Physical Volume create) on each drive, adding them to a Volume Groug (VG). I realized that the drives could be categorized as Slow and Fast, depending on their seek times (and whether they “fell asleep” between access). So I added the PVs into big and big2 groups [using VGcreate](not the best naming, should have used Fast and Slow).

sudo vgs 
VG   #PV #LV #SN Attr   VSize   VFree  
 big    5  28   0 wz--n-  41.84t 11.36t
 big2   3   9   0 wz--n- <22.74t <5.99t

Finally I started creating Logical Volumes (LVcreate) for each purpose always creating LVcreate on VG1 and VG2 same size, and creating an rsync between the LVs (backup of a backup — effectively bring the whole array into a RAID1 configuration). Now, I could have used LVM2s native tools to do this as well, but I wanted more control (and I had been burned by LVM1 in the pre-historic 10year+ past).

Nowadays, I have 38 Logical volumes (spread across 9 PVs), that I occasionally expand or shink depending on size needed, usually about a terabyte at a time.

My LVS map is too long (and private to show).

Here’s the killer!
When the California heatwave occured, I lost the machine (srv3) completely. and I built a new server (srv4), I just took all the USB hubs from srv3, plugged into srv4, and ran PVSCAN. LVM2 rebuilt the whole raid array, with configurations (PVS, and VGs). All I needed to do was copy the /etc/fstab from a backup of srv3… and we’re back in business!

The USBs, the drive IDs, the HUBs don’t matter, everything fell back in place correctly.

The only thing I wish is that I hoped I hade done better job in taking mysqldumps of the databases instead of doing rsyncs, thus not needing to rebuild the sites. But the sites that were HTML came back immediately.

I had a bad USB hub, causing some PVs (Physical Volumes) to go off-line, and since I have many of the Logical Volumes shared/mounted via NFS that became a problem. Here is a script that 1) takes NFS off-line, 2) checks all the mount points, 3) re-mounts them, 4) restarts NFS. Easy to run, All shares restored, all volumes re-mounted.

cat /root/reseatusb.sh  
#/bin/bash

sudo /etc/init.d/nfs-kernel-server stop
DF1=$(df -h|grep /dev/mapper|awk '{print $1}')
DF6=$(df -h|grep /dev/mapper|awk '{print $6}')
echo $DF6|xargs umount
echo $DF1|xargs lvchange -an  
echo $DF1|xargs lvchange -ay  
mount -a
sudo /etc/init.d/nfs-kernel-server start

Of course, once I realized it was because the bad hub, I replaced with another USB3 hub, and I’ve never needed the script anymore.

Emails on user accounts were a bit trickier, Folder by folder reconstruction became a possibility, though some tricks were involved (sendmail, dovecot — I will probably write another article). I will likely add more details to this if interested.

Leave a Reply