LVM – Adams Bros Blog http://blog.adamsbros.org Wed, 15 May 2019 01:42:42 +0000 en-US hourly 1 https://wordpress.org/?v=5.2.2 Making New Linux Disk Visible Without a Reboot http://blog.adamsbros.org/2015/11/16/making-new-linux-disk-visible-without-a-reboot/ http://blog.adamsbros.org/2015/11/16/making-new-linux-disk-visible-without-a-reboot/#respond Mon, 16 Nov 2015 19:28:56 +0000 http://blog.adamsbros.org/?p=574 I was having trouble today getting Linux to see my new partition space that I added in vSphere without rebooting the host. The new disk space was made visible by re-scanning the SCSI bus (below) and then the new partition was made visible by using the partprobe command (below).

 

I asked VMWare to provision my disk to be larger and then asked Linux to refresh the kernel info:

 $ echo 1 > /sys/class/scsi_device/0\:0\:3\:0/device/rescan 
 $ dmesg
 sdd: Write Protect is off 
 sdd: Mode Sense: 61 00 00 00 
 sdd: cache data unavailable 
 sdd: assuming drive cache: write through 
 sdd: detected capacity change from 171798691840 to 343597383680

 

I added another partition and then tried to get LVM to use it:

$ fdisk /dev/sdd
 Command (m for help): n
 Command action
 e extended
 p primary partition (1-4)
 p
 Partition number (1-4): 3

But LVM couldn’t see it:

 $ pvcreate /dev/sdd3
 Device /dev/sdd3 not found (or ignored by filtering).
 $ pvcreate -vvvv /dev/sdd3
 #device/dev-cache.c:578 /dev/sdd3: stat failed: No such file or directory
 #metadata/metadata.c:3546 
 #device/dev-cache.c:578 /dev/sdd3: stat failed: No such file or directory


The solution was to use partprobe to inform the OS of partition table changes:


 $ partprobe /dev/sdd
 $ pvcreate /dev/sdd3
 dev_is_mpath: failed to get device for 8:51
 Writing physical volume data to disk "/dev/sdd3"
 Physical volume "/dev/sdd3" successfully created

]]>
							http://blog.adamsbros.org/2015/11/16/making-new-linux-disk-visible-without-a-reboot/feed/
		0
							
		
		Enumerate All Block Devices
		http://blog.adamsbros.org/2014/12/07/enumerate-all-block-devices/
				http://blog.adamsbros.org/2014/12/07/enumerate-all-block-devices/#respond
				Sun, 07 Dec 2014 11:04:40 +0000
		
				
		
		

		http://blog.adamsbros.org/?p=560
				
								I found a great little utility that can enumerate through all the block devices, including your lvm, crypt, etc.




It’s found in the util-linux-ng package, and is called lsblk.  Here’s an example run…


[04:04:06 root@tdanas ~]
$ lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0 111.8G  0 disk  
├─sda1                    8:1    0  1000M  0 part  /boot
└─sda2                    8:2    0 110.8G  0 part  
  ├─nas-root (dm-0)     253:0    0  19.5G  0 lvm   /
  ├─nas-swap (dm-1)     253:1    0  1000M  0 lvm   [SWAP]
  ├─nas-log (dm-3)      253:3    0   9.8G  0 lvm   /var/log
  └─nas-tmp (dm-4)      253:4    0   9.8G  0 lvm   /tmp
sdb                       8:16   0   2.7T  0 disk  
├─sdb1                    8:17   0 931.3G  0 part  
│ └─md10                  9:10   0   500G  0 raid1 
│   └─pv0 (dm-5)        253:5    0   500G  0 crypt 
│     └─dat-home (dm-6) 253:6    0   500G  0 lvm   /data
└─sdb2                    8:18   0   1.8T  0 part  
sdc                       8:32   0 931.5G  0 disk  
└─sdc1                    8:33   0 931.5G  0 part  
  └─md10                  9:10   0   500G  0 raid1 
    └─pv0 (dm-5)        253:5    0   500G  0 crypt 
      └─dat-home (dm-6) 253:6    0   500G  0 lvm   /data
sdd                       8:48   0   2.7T  0 disk  
└─sdd1                    8:49   0   2.7T  0 part  
  └─3TBak_crypt (dm-7)  253:7    0   2.7T  0 crypt 
    └─3TB-safe (dm-8)   253:8    0   1.5T  0 lvm   /media/cvsafe
sde                       8:64   0 931.5G  0 disk  
└─sde1                    8:65   0   500G  0 part  
  └─md10                  9:10   0   500G  0 raid1 
    └─pv0 (dm-5)        253:5    0   500G  0 crypt 
      └─dat-home (dm-6) 253:6    0   500G  0 lvm   /data



You can also specify the ‘-f’ flag, if you want UUIDs of the block devices for recovery information.

]]>
							http://blog.adamsbros.org/2014/12/07/enumerate-all-block-devices/feed/
		0
							
		
		Safe Shrinking of ext3 LVM volumes
		http://blog.adamsbros.org/2010/01/11/safe-shrinking-of-lvm-volumes/
				http://blog.adamsbros.org/2010/01/11/safe-shrinking-of-lvm-volumes/#comments
				Mon, 11 Jan 2010 08:53:55 +0000
		
				
		

		http://blog.adamsbros.org/?p=102
				
								When shrinking your LVM volumes, it is important to do it safely.  I will show you how I like to do that here.


Shrinking


Shrinking an ext3 volume is fairly straight forward, but it’s important you don’t mount your drive and start doing stuff, before you verify everything worked correctly.


    

  1. e2fsck -f /dev/vg/lv1
    2. 

  2. resize2fs -p /dev/vg/lv1 ###G
    3. 

  3. e2fsck /dev/vg/lv1
    4. 

  4. lvresize -L ###G /dev/vg/lv1
    5. 

  5. e2fsck /dev/vg/lv1
    6. 



Step #5 is the most important step; as steps 1-3 will not have any problems unless you have a system failure, a corrupted file system to start with, disk failure, or hardware failure.  Step 5 however, could be due to sys-admin error, from shrinking the LVM volume too much.  If you receive an error similar to the following, then you probably shrunk the volume too much, and your file system is not completely accessible.




e2fsck 1.41.9 (22-Aug-2009)
The filesystem size (according to the superblock) is 113634135 blocks
The physical size of the device is 112197632 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? cancelled!




If the error is due to shrinking too far, then this is easy to recover from; simply do another lvresize, with a large size specified, and do another e2fsck afterward.


Generally speaking, you should make the LVM maybe 2% or so larger than the file system was reduced to, so that there’s room for file system overhead.

]]>
							http://blog.adamsbros.org/2010/01/11/safe-shrinking-of-lvm-volumes/feed/
		1
							
		
		Recover LVM Volume Groups and Logical Volumes WITHOUT Backups
		http://blog.adamsbros.org/2009/05/30/recover-lvm-volume-groups-and-logical-volumes-without-backups/
				http://blog.adamsbros.org/2009/05/30/recover-lvm-volume-groups-and-logical-volumes-without-backups/#comments
				Sat, 30 May 2009 08:49:10 +0000
		
				
		

		http://blog.adamsbros.org/?p=7
				
								I recently had a misfortune, in that somehow my volume group meta-data got corrupted, and LVM would not enable the volume group. Essentially, I lost my LVM volume disk. This happened after I resized a volume, and had done a file system check before and after. So, I knew my data was still there.


I did an lvextend on my primary logical volume. Normally this is a routine task, but for some reason, things went very badly for me this time around. I did an “fsck -f” before and after extending the volume and the filesystem (with resize2fs). Everything checked out just fine, so I thought everything was done, and ready to reboot.


I proceeded to issue the three finger salute, and my system began the reboot process. Upon trying to boot up, I got errors about my root logical volume not being found. So, I booted up with a gentoo live cd again, and got the following errors…




root@Microknoppix:~# pvscan
/dev/hda: open failed: Read-only file system
Attempt to close device '/dev/hda' which is not open.
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Volume Group s is not consistent
PV /dev/sdb5   VG bak             lvm2 [32.00 GB / 0    free]
PV /dev/sdb6   VG bak             lvm2 [266.09 GB / 0    free]
PV /dev/sda4   VG s               lvm2 [207.58 GB / 88.70 GB free]
PV /dev/sdf2                      lvm2 [59.88 GB]
PV /dev/sdf4                      lvm2 [88.89 GB]
Total: 5 [654.42 GB] / in use: 3 [505.66 GB] / in no VG: 2 [148.76 GB]

root@Microknoppix:~# vgchange -ay
Incorrect metadata area header checksum
Incorrect metadata area header checksum
1 logical volume(s) in volume group "bak" now active
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Volume group "s" inconsistent
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Inconsistent metadata found for VG s - updating to use version
19
Incorrect metadata area header checksum
Automatic metadata correction failed




So, what to do? I didn’t have a system level backup (just data), because I hadn’t gotten a round to it yet, after converting from Mac OS X. Well, with gentoo Linux, this is a bit devastating, because you have to recompile entirely from source. That can take days, and sometimes it takes months to figure out all the settings you had, if you don’t have a backup. To boot, I didn’t even have a backup of ‘/etc/’, where the LVM backups are stored, DOH!!! As a result, all the lvm backups were unavailable. This is all really sad, seeing I’m an automatic backup freak, and can’t stand it when I don’t have system backups.


Now, one could go searching for the beginning of their logical volume, if they wanted to, and recover just that. But, that could be very painful, especially if your LV is not contiguous; which can easily happen if you have multiple drives in your volume group, and you have been resizing your volumes a few times.


Well, as it goes, my brother suggested I boot up with a knoppix CD, to see if I could figure out how to fix the problem. I began running a few different commands. I started with pvs, which displays physical volume information.




root@Microknoppix:~# pvs -v

Scanning for physical volume names
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Volume Group s is not consistent
Incorrect metadata area header checksum
Incorrect metadata area header checksum
PV         VG   Fmt  Attr PSize   PFree  DevSize PV UUID
/dev/sda4  s    lvm2 a-   207.58G 88.70G 207.58G 8cYXSr-l35B-2HBg-V7YS-TWsb-rZ8L-C5EC7J
/dev/sdb5  bak  lvm2 a-    32.00G     0   32.00G Vx3gVW-YNoq-xLHt-rOaJ-2HHW-qBcj-nJSfmx
/dev/sdb6  bak  lvm2 a-   266.09G     0  266.09G o7Mi6k-lEsH-ndqb-QMxe-3t3Z-jTWS-qw9KKv




Next I did “vgdisplay -v” dump, and then I ran into pvck by accident. This will display your physical volume information. It happens to display the offsets to all of your LVM metadata backups (GREAT).  I’ve heard that these are stored in a cycling manner.  So, it may not be worth paying attention to the order they appear.




root@Microknoppix:/mnt/safe/trenta# pvck -d -v /dev/sda4
Scanning /dev/sda4
Incorrect metadata area header checksum
Found label on /dev/sda4, sector 1, type=LVM2 001"
Found text metadata area: offset=4096, size=192512
Found LVM2 metadata record at offset=26624, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=24576, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=22528, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=20480, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=17920, size=2560, offset2=0 size2=0
Found LVM2 metadata record at offset=15360, size=2560, offset2=0 size2=0
Found LVM2 metadata record at offset=12800, size=2560, offset2=0 size2=0
Found LVM2 metadata record at offset=10752, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=8704, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=6656, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=4608, size=2048, offset2=0 size2=0
Found LVM2 metadata record at offset=30720, size=165888, offset2=0 size2=0
Found text metadata area: offset=96646856704, size=183296
Incorrect metadata area header checksum




At this point, I’m becoming a little more cheery; I have a chance at recovery, FINALLY; no recompiling my entire system. So, I check knoppix to find out if it has a hex editor, and sure enough, there is an editor called hexedit. So, I ran “hexedit /dev/sda4”. I converted the offsets from the pvck output to hex, and went to those offsets by paging down in hexedit. The last record is 30720, which equates to 0x7800. I start highlighting at that offset (Ctrl-Space), and then select the entire file up until the 0x0A0A newline bytes at the end of it. I copy (Esc-W), and then I paste to a file (Esc-Y), and call it /path/lvm-metadata-30720.txt. I do this for everyone that I think might contain information I need. In order to know whether it’s relevant data or not, you have to know what the last few LVM changes you made were. Things like sizes of logical volumes, the size of the volume group, what physical volumes existed in the volume group, and things of that nature will all be helpful in recovery. For example, did you double the size of a logical volume, or reduce, or whatever? Then, it’s just a matter of diffing each version, to see when the important changes were made. My diff shows…




root@Microknoppix:/mnt/safe/trenta# diff -u lvm-metadata-26624.txt lvm-metadata-28672.txt
--- lvm-metadata-26624.txt      2009-05-25 23:45:11.000000000 +0000
+++ lvm-metadata-28672.txt      2009-05-25 23:41:22.000000000 +0000
@@ -1,6 +1,6 @@
s {
id = "HNHKOr-RpyA-uMdz-tqhN-Z673-L2ej-qXikhF"
-seqno = 18
+seqno = 19
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 8192
max_lv = 0
@@ -64,7 +64,7 @@

segment1 {
start_extent = 0
-extent_count = 7680
+extent_count = 15360

type = "striped"
stripe_count = 1       # linear
@@ -94,7 +94,7 @@
}
}
}
-# Generated by LVM2 version 2.02.36 (2008-04-29): Mon May 25 02:42:45 2009
+# Generated by LVM2 version 2.02.36 (2008-04-29): Mon May 25 02:44:25 2009

contents = "Text Format Volume Group"
version = 1
@@ -102,5 +102,5 @@
description = ""

creation_host = "tdamac"       # Linux tdamac 2.6.30-rc3-dirty #23 SMP Mon May 25 01:39:04 MDT 2009 x86_64
-creation_time = 1243240965     # Mon May 25 02:42:45 2009
+creation_time = 1243241065     # Mon May 25 02:44:25 2009




If you look carefully in my diff, you will see the seqno changed from 18 to 19, and the size of the “segment” doubled; that is what I had just done as part of my logical volume resize. The “vgchange -ay” command previously tried to restore seqno=19, which happened to fail. As it were, it appears that v19 of the meta-data backups is the version I need. So, now it’s just a matter of taking all of the data we have dumped, and recreating the LVM information. It may be worth doing a RAW dd dump of your partition before messing with it, but I didn’t bother. I was fairly certain that my data was still there, and I knew that restoring a configuration would not wipe out the data, just the LVM meta-data.


So, finally I completely recreated everything by doing the following.  The UUID for the physical volume comes from the meta data file.  It’s important that you choose the correct ID, or it won’t match the LVM data for the vgcfgrestore.




root@Microknoppix:~# pvcreate -ff -u 8cYXSr-l35B-2HBg-V7YS-TWsb-rZ8L-C5EC7J \
  --restorefile lvm-metadata-28672.txt /dev/sda4
root@Microknoppix:~# vgcfgrestore -f /mnt/safe/trenta/lvm-metadata-28672.txt -v s




I am pretty certain that only the second command was needed, because the physical volume meta-data was fine, it was the volume group that was messed up. But, I was hacking my way through, so I issued both commands.


WARNING, WARNING, WARNING, if you do this wrong, and use the wrong meta-data information, you may overwrite some file data somewhere else on the disk.  This can happen when the extents and sizes are off, so the vgcfgrestore command restores the meta-data to the wrong parts of the disk.  Be sure you are able to pick the correct backup data to restore, or you very well may loose data.


Now, after all of this, I think it may have been as easying as doing the following, but I am unsure.  I think what would happen with the following command, is that it would ignore the locking failure, and restore the configuration regardless.




vgchange -ay --ignorelockingfailure




Either way, this information is useful in the event that you loose meta-data due to bad blocks or whatever.


 


    

  • how to repair a logical volume
    • 

  • how to repair bad configured lvm
    • 

  • incorrect metadata area header checksum on
    • 


]]>
							http://blog.adamsbros.org/2009/05/30/recover-lvm-volume-groups-and-logical-volumes-without-backups/feed/
		26