setting up software raid zero over 8 volumes on EC2 using EBS, mdadm and xfs

I’ve read a lot of blog posts about how to set up software raid on EC2 using EBS volumes but didn’t find all the steps in one page, this is my attempt to do so.

Before you start you will need to have the ec2-api-tools installed on your local machine which can be found here:

http://aws.amazon.com/developertools/351?_encoding=UTF8&jiveRedirect=1

I won’t go into the steps for installing and configuring but I will list the local envorionment variables that you should have in your .bash_profile

They are:

export PATH=$PATH:/Users/dodell/ec2-api-tools/bin
export EC2_HOME=’/Users/dodell/ec2-api-tools’
export JAVA_HOME=’/System/Library/Frameworks/JavaVM.framework/Home’
export EC2_CERT=/Users/dodell/aws_keys/cert-XXXXXXXXXXXX.pem
export EC2_PRIVATE_KEY=/Users/dodell/aws_keys/pk-XXXXXXXXXXXXXXX.pem

In this post I’m detailing a raid 0 volume I created which is 4TB total in size and striped over 8 EBS volumes.

***TRICK***

Also one really helpful trick I found was to tag the volumes you create with the host they will be attached to and the device they will be attached as.

After tagging its really easy to tell which volume gets attached to which host

ec2-describe-volumes vol-XXXXXXX
VOLUME    vol-XXXXXX    500        us-east-1a    in-use    2011-01-11T23:49:19+0000
ATTACHMENT    vol-XXXXXXX    i-XXXXX    /dev/sdm    attached    2011-01-11T23:58:44+0000
TAG    volume    vol-XXXXXXX   host    mdb05
TAG    volume    vol-XXXXXXX   mount    sdm

***TRICK***

Steps 1 -4 happen on your local machine

Step 1 – create the volumes

ec2-create-volume -z us-east-1a -s 500

This will create a 500GB volume
Repeat as necessary

Step 2 – tag the volumes with which host they will be attached to

ec2-describe-volumes | grep '2011-01-11T23' | grep available| awk '{print $2}'| ec2-create-tags - -t host=mdb05

This will apply the same tag to all the volumes you created above( you will need to change what you grep for )

Step 3 – attach the volumes to the host

ec2-attach-volume vol-XXXXXX -i i-XXXXX -d /dev/sdm

repeat as necessary

Step 4 – tag the volume with which device it will be mounted as

ec2-create-tags vol-XXXXXX -t mount=sdm

repeat as necessary


Now you have x number of volumes attached to your instance.



Steps 5 – 13 occur on the instance where you attached the volumes


Step 5 – create the raid device using mdadm

sudo mdadm --create /dev/md0 --chunk=256 --level=0 --raid-devices=8 /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt

Step 6 – verify that the raid device /dev/md0 exists

cat /proc/mdstat

Step 7 – add devices to the mdadm.conf file

sudo echo DEVICE /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt| sudo tee /etc/mdadm/mdadm.conf

Step 8 – add the other device info about /dev/md0 to the mdadm.conf file so that it comes back on reboot

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Step 9 – make the file system

sudo mkfs.xfs /dev/md0

Step 10 – mount the file system

sudo mount /dev/md0 /data/

Step 11 – verify the volume exists and the size you expected

df -h /data
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.0T  250G  3.7T   7% /data

Step 12 – edit fstab so that the volume comes back on reboot

/dev/md0    /data    xfs    noatime,nodiratime,allocsize=512m      0       0

Step 13 – test all of you work by rebooting

reboot