Case Study: Backup Device LVM issue

Background:

A Client recently moved to Idera backup for their mail server (Exchange) when the previous Symantec backup stopped working after a P2V conversion.  These backups were successful, but when the backup device was moved to a new location and powered on, it was unreachable.

 


Problem Being Addressed:

It was presumed that the Backup Device (which was not given a static IP) simply needed to have the networking reconfigured.  However, upon arrival, the device did not show up in the DHCP lease page on the DHCP server.  This prompted further investigation of the system.  When a keyboard and monitor was connected, it revealed that the device was unable to boot, with an error about being unable to mount key filesystems, and a prompt to enter maintenance mode.

boot prompt

The Approach Taken:

Initial basic troubleshooting was conducted (reconnect SATA cables, reboot the machine) but all drives were securely connected and identified in the BIOS.  All drives passed SMART tests.  IDE vs AHCI BIOS options made no difference.  Logging in to maintenance mode and reviewing the error messages pointed to /dev/mapper/ entries that could not complete.  Examining this location revealed that the triggered entries no longer existed.  The offending devices were removed from /etc/fstab (the configuration file that handles mapping drives on boot) and the system booted – though now without the /home and /data directories accessible (as these were on the “missing” drives).


Problems Experienced:

When further examining the problem it was determined that the system was originally set up using LVM (Logical Volume Manager), which is a linux system that pools all Physical Volumes (PVs) into Volume Groups (VGs), which can then be divided into Logical Volumes (LVs) that are exposed to the OS as disks, and can be mounted to a location.  Because the system was set up incorrectly (LVM is okay for temporary use, but offers no protection of data and can increase unreliability) it was determined that the system should be reinstalled.  However, we needed to extract the backup data from the current machine before reformatting in case there was a failure of the current system before new backups could begin again.

Normally the backup data would have been located in the /data directory (as per the original setup).  However, this directory was inaccessible.  By using LVM commands, it was determined that all of the LVM setup information was present, but the auto-mounting of the volume was the failing component.  When LVM tools are installed on a system, you can use the commands pvs, vgs, and lvs to see all of the information.  The lvs command showed that the two Logival Volumes (fedora-home and fedora-data) were configured, but not active.  The commands “lvchange –ay /dev/fedora-home” and “lvchange –ay /dev/fedora-data” activated these devices so they could be used.  Next, using “mount /dev/fedora-home /home” and “mount/dev/fedora-data /data” mounted the files to their correct locations so that the data inside was accessible, and could be copied to an external hard drive.


Things We Would Do Differently:

Ultimately, the device should have been configured differently from the start.  As previously stated, LVM is not suitable for long-term use, and instead standard partitions should be created and mapped directly (mount /dev/sda /data, for example).


References:

  1. lvchange(8) – Linux man page
  2. mount(8) – Linux man page


Inactive since April, 2015

Andrew Sherman graduated from Linfield College in 2011 with a Bachelors in Psychology and a minor in business. He has worked with computers from a very young age, which led to his first full IT job in High-school. During college, he worked for Linfield’s IT department, and continued developing skills with computer deployment and troubleshooting.

Since joining Deus Machine Andrew has honed a large array of IT skills including Network Administration and Engineering, as well as Windows Server, Active Directory, and Microsoft Exchange Deployment and administration. He specializes in configuration of network hardware, and is the go-to Cisco specialist in the company.