Setting Up LVM On Top Of Software RAID Subsystem – RHEL & Fedora

Many Linux users may well know about the Logical Volume Management in Linux and the software RAID subsystem. The Logical Volume Manager (LVM) is a mechanism for virtualizing disks. It creates “Virtual disk drives and volumes(partitions)” out of one or more physical hard drives, allowing users to grow, shrink, move those VOLUMES from drive to drive as needed. In a Logical Volume Set, Logical Volumes are easy to resize without the need for repartitioning, backing up or losing the DATA whereas RAID provides a Fault Tolerant Mechanism for Disk crashes. Uses of LVM include extensively larger databases and file servers, but even home users may want large partitions for storing large amount of media files, etc.

You may find another very familiar situation, say like you are the System administrator of a company having 500 employees. All 500 employees use a centralized login server (NIS). Let this NIS server has home directory of 500 employees as /home/nisuserX. On a 400GB drive and /home of 300GB. Suppose with time, this entire 300GB fills out and you need to add extra storage space to /home, and preserving users data on the /home at the same time. If you had done a Normal partitioning setup at the start, this is not easy or possible. Also keeping data on a Single Drive is very risky. We need to safeguard from disk failures too. But “Simply implementing LVM”, has one major flaw. If any of the drives fail, all of your data is at risk. To compensate for this risk, we will build redundancy into LVM using RAID 1(Mirroring) technique in our example. But you can also try out RAID 5 and 6. Another practical situation can be a Mail Server storing increasing email users and their respective data.

Here is a quick look how to build an LVM on top of RAID 1 array, so that we combine the power of these two. This kind of setup is extremely useful in situations where we want a file server to store large amounts of data which provides a centralized backup, storage space for downloadable files via ftp/http, and that may grow enormously in the coming years after the initial setup.

Our example setup will be on a single hard disk (SATA) and FEDORA 8/REDHAT Enterprise Server Linux 5, so that all Linux users can practice this configuration even on a single disk with software raid and lvm support. The following configurations have been successfully tested on an Acer4710z Laptop with 160GB HDD, 1GB RAM, 1.73GZ CPU and Fedora 8 OS.

As u may assume that we will setup raid first and then lvm on top of it, instead I will first create the lvm and then raid and do little swapping between the two to get lvm on top of raid subsystem.


This example setup has the following configs:-

160gb SATA HDD & Fedora 8 OS installed on it .

/dev/sda1  as /boot=110MB

/dev/sda2  as /=57GB

/dev/sda3 as swap= 2GB

/dev/sda4 (extended) =100GB

Free space on the disk is around 100gb.Each physical volume (PV) is of 2GB.VG=4 GB, LV=1GB and “RAID 1” array of 2GB. Here are the necessary steps for the setup:

Step 1:

Log in as “root”. Create Two partitions (sda5, sda6) of type LVM (8e) on the Primary Master SATA Drive.

# fdisk /dev/sda

Type : n (for new partition)

Type: l (logical), leave the first cylinder blank, press enter

Type : +2048M as the last cylinder value, press enter.

Type : t (change the partition ID)

Type: 5 (to select /dev/sda5)

Type : 8e (LVM type)

Repeat the steps to create /dev/sda6 of 2 GB size also. Then type “w” to save.

Step 2:

# partprobe

Step 3:

Create the PHYSICAL VOLUMES on /dev/sda5 and /dev/sda6.

# pvcreate /dev/sda6 /dev/sda6

Step 4:

Now create the VOLUME GROUP (VG) out of only /dev/sda5. Also create the VG node file in /dev.

# vgcreate vg1 /dev/sda5

# vgmknodes vg1 (optional)

Note: vgmknodes vg1 tries to remove /dev/vg1 and all contained special files and then creates them again based on the VGDA.

Step 5:

Create LV of 1GB size, format and mount it.

# lvcreate –L 1024M –n lv1 vg1

# mkfs.ext3 /dev/vg1/lv1

Create a mount point.

# mkdir /lv

# mount /dev/mapper/vg1-lv1 /lv (or) # mount /dev/vg1/lv1 /lv

Step 6:

To mount it permanently on boot, make the entry in /etc/fstab file.

# vi /etc/fstab

/dev/mapper/vg1-lv1 /lv ext3 defaults 1 2

Esc : wq, press enter to save the file.

Step 7:

Now extend the volume group (vg1) by adding PV /dev/sda6 to it.

# vgextend vg1 /dev/sda6


Move the physical extents on PV /dev/sda5 to /dev/sda6 and pull /dev/sda5 out of VG.

# pvmove /dev/sda5 /dev/sda6

# vgreduce vg1 /dev/sda5

# pvremove /dev/sda5

Step 9:

Using fdisk, change the partition ID of /dev/sda5 from 8e to fd (raid type), save by typing “w” in the fdisk menu.

then do:

# partprobe

Step 10:

Create the RAID ARRAY with /dev/sda5 and 1 missing device (/dev/sda6). Remember that RAID 1 requires a min. of 2 active devices, we will add /dev/sda6 later.

# mdadm – -create /dev/md0 – -level 1 –n 2 /dev/sda5 missing

Step 11:

Create PV on /dev/md0 and add it to the VG vg1.

# pvcreate /dev/md0

# vgextend vg1 /dev/md0


Then move the Physical Extents on PV, /dev/sda6 to /dev/md0 and pull /dev/sda6 out of VG.

# pvmove /dev/sda6 /dev/md0

# vgreduce vg1 /dev/sda6

# pvremove /dev/sda6

Use pvdisplay, vgdisplay and lvdisplay to get the status of PV’s, VG and LV’s.

Note the size of /dev/md0from the output of pvdisplay.


Using fdisk change the partition type ID of /dev/sda6 to fd (raid), save by “w” and then do

# partprobe

Step 14:

Now add the /dev/sda6 to the RAID array /dev/md0.

# mdadm – -manage /dev/md0 – -add /dev/sda6

# pvresize /dev/md0 (Resize the PV on /dev/md0)

# mdadm – -grow /dev/md0 –z max (Refresh the Array)

The GROW mode is used for changing the size or shape of an active array. ” –z max” means, amount (in Kilobytes) of space to use from each drive in /dev/md0. This value is set with –grow for RAID l since, the array was created with a size smaller than the currently active drives. Now the extra space can be accessed using –grow.

Use pvdisplay, vgdisplay and lvdisplay to get the status of PV’s, VG and LV’s. Can you notice the changes in the output of the pvdisplay command?

Step 15:

Get the details of /dev/md0 and add to /etc/mdadm.conf. First copy the mdadm.conf.sample file to the /etc directory. This step is usefull to rectify situations where after creating RAID array and rebooting the system, you get error saying “no /dev/md0 found”.

# cp /usr/share/doc/mdadm*/mdadm.conf-sample /etc/mdadm.conf

# mdadm – -detail /dev/md0

Note down the UUID of the raid array.

# vi /etc/mdadm.conf

And add these two lines:

DEVICE /dev/sda5 /dev/sda6

ARRAY /dev/md0 UUID=<ID>

Save the file using Esc:wq.

Then do:

# mdadm – -detail /dev/md0

So we have seen that the LVM (LV-lv1) sits on top of RAID 1 array, called the LVM mirrored set. This method of configuration does not need to unmount the LV and also prevents data loss. We will further go one step ahead and see how to add a spare drive if a device in the array fails or becomes faulty.

Open a new terminal (ctrl+alt+F2) and type:

# watch cat /proc/mdstat (to see the live status of RAID array /dev/md0)

Make /dev/sda6 faulty and then remove it from the raid array.

# mdadm – -manage /dev/md0 – -fail /dev/sda6 – -remove /dev/sda6

Create a new partition of type fd. Here, /dev/sda7.

# fdisk /dev/sda n-> l-> +2048M (/dev/sda7)-> t ->7 -> fd -> w

# partprobe

Add the /dev/sda7 device to the raid array:

# mdadm – -manage /dev/md0 – -add /dev/sda7

# vi /etc/mdadm.conf

Change the DEVICE line: DEVICE /dev/sda5 /dev/sda7

Save the file using Esc:wq.

Then do:

# mdadm – -detail /dev/md0

# pvresize /dev/md0

# mdadm – -grow /dev/md0 –z max

This will refresh and resize the PV and RAID array respectively.


Conclusion: You can test this setup with different Physical Hard Disks. Combing LVM and RAID gives more Flexibility as well as safeguards the data to great extent.



Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: