A Client recently moved to Idera backup for their mail server (Exchange) when the previous Symantec backup stopped working after a P2V conversion. These backups were successful, but when the backup device was moved to a new location and powered on, it was unreachable.
Problem Being Addressed:
It was presumed that the Backup Device (which was not given a static IP) simply needed to have the networking reconfigured. However, upon arrival, the device did not show up in the DHCP lease page on the DHCP server. This prompted further investigation of the system. When a keyboard and monitor was connected, it revealed that the device was unable to boot, with an error about being unable to mount key filesystems, and a prompt to enter maintenance mode.
The Approach Taken:
Initial basic troubleshooting was conducted (reconnect SATA cables, reboot the machine) but all drives were securely connected and identified in the BIOS. All drives passed SMART tests. IDE vs AHCI BIOS options made no difference. Logging in to maintenance mode and reviewing the error messages pointed to /dev/mapper/ entries that could not complete. Examining this location revealed that the triggered entries no longer existed. The offending devices were removed from /etc/fstab (the configuration file that handles mapping drives on boot) and the system booted – though now without the /home and /data directories accessible (as these were on the “missing” drives).
When further examining the problem it was determined that the system was originally set up using LVM (Logical Volume Manager), which is a linux system that pools all Physical Volumes (PVs) into Volume Groups (VGs), which can then be divided into Logical Volumes (LVs) that are exposed to the OS as disks, and can be mounted to a location. Because the system was set up incorrectly (LVM is okay for temporary use, but offers no protection of data and can increase unreliability) it was determined that the system should be reinstalled. However, we needed to extract the backup data from the current machine before reformatting in case there was a failure of the current system before new backups could begin again.
Normally the backup data would have been located in the /data directory (as per the original setup). However, this directory was inaccessible. By using LVM commands, it was determined that all of the LVM setup information was present, but the auto-mounting of the volume was the failing component. When LVM tools are installed on a system, you can use the commands pvs, vgs, and lvs to see all of the information. The lvs command showed that the two Logival Volumes (fedora-home and fedora-data) were configured, but not active. The commands “lvchange –ay /dev/fedora-home” and “lvchange –ay /dev/fedora-data” activated these devices so they could be used. Next, using “mount /dev/fedora-home /home” and “mount/dev/fedora-data /data” mounted the files to their correct locations so that the data inside was accessible, and could be copied to an external hard drive.
Things We Would Do Differently:
Ultimately, the device should have been configured differently from the start. As previously stated, LVM is not suitable for long-term use, and instead standard partitions should be created and mapped directly (mount /dev/sda /data, for example).