Category: virtual-machines

So I’ve got a bit of a situation, I’ll try to keep it brief!
Initial server setup:

2x drives with soft RAID1
LVM thin-pool setup on the md3 RAID device

Then the md3 (raid) device gets full so in my haste to fix the problem (knowing I can’t get an additional drive added at the DC) I removed the 2nd disk from RAID and setup a new PV on it. The remaining disk was left in RAID (just 1 drive) as a PV on the VG. I then extended the thin-pool to this “new” PV to effectively double the storage within the thin-pool for use by the VMs.
The mistake I made when doing this is (probably one of many), I forgot to completely format the drive so the previous LV structure was still present when I extended the thin-pool but I didn’t realise this until some time afterwards. However from my research I understand that when you set up a new PV on an existing volume, it’s a destructive act so despite the previous LV structure still showing the new drive is being used by LVM and it is spanning any new data across the two PV’s.
Not a perfect solution but it resolved the initial issue of no space being available.

Now I’m left trying to resolve this issue:
The thin-pool across the two drives is only 43% full of data + meta data

[root@XXX ~]# lvs
  LV                                        VG  Attr       LSize   Pool      Origin Data%  Meta%  Move Log Cpy%Sync Convert
  test                                      vg1 Vwi-a-tz--   5.00g thin_pool        20.00
  thin_pool                                 vg1 twi-aotz--   3.14t                  42.63  28.69

however…

[root@XXX ~]# pvdisplay -C
  PV             VG  Fmt  Attr PSize  PFree
  /dev/md3       vg1 lvm2 a--  <1.58t     0
  /dev/nvme0n1p4 vg1 lvm2 a--  <1.58t 15.73g
[root@XXX ~]# vgdisplay -C
  VG  #PV #LV #SN Attr   VSize  VFree
  vg1   2  59   0 wz--n- <3.16t 15.73g

It would appear that even though there should be enough space left across the two drives, the PE’s that were “left over” (when I forgot to format) are still being treated as if they’re in use so I barely have any free PE’s left.

When you do a lvs the LVs are not duplicated in either quantity or LV size, they’re exactly as I would expect them to be and the usage of the LV’s again is what I would expect them to be, not duplicated or doubled.

Any ideas on how to resolve this highly frustrating and self-made situation?

Any help would be enormously appreciated.