Adding XFS support to 64-bit SL/SL4.x for DPM Disk Server

Hardware / OS

• System :: Dual-Core AMD Opteron 2210, 4GB of memory
• RAID :: 3Ware/AMCC 9650SE-16M [RAID6]
• Storage :: ~7.5TB per unit (usable), configured as 13-disk RAID6 plus hot spare.

  (Could be 14-disk since RAID6, but we prefered the hot spare configuration)

• OS :: SL-4.5 [ x86_64 ]
• Installation :: Kickstart and manual combined

Problem-1

RHEL4, update 5 only provides the in-built support for 3Ware 9650SE RAID controller. So any version below 4.5 (SL or SLC) simply won't work out of the box. There are some drivers available for selected lower version of SL/SLC on the 3Ware/AMCC website. We started with SL4.5 just make the installation as simple as possible.

Problem-2

It would appear that fdisk has a 2TB partition limit and the partition table labeled as msdos or BSD also doesn't work for a partition larger that 2TB, hence post-installation boot eventually fails due a misconfigured “fstab” file. As we wanted to have a single ~7.5TB partition on each unit for DPM file system, we initially excluded the disk array from the kickstart configuration file and formated afterwards the kickstart installation.

OS Installation

We started with normal kickstart installation but excluding /dev/sdb (which is actually the 7.5TB disk array) and with the following packages.

# Package install information
%packages --resolvedeps
@ Editors
@ Text-based Internet
@ Development Tools
@ Administration Tools
@ System Tools
@ yum
kernel
kernel-smp
kernel-devel
kernel-smp-devel
e2fsprogs
kernel-smp-devel
grub

Just make sure to install kernel-devel/kernel-smp-devel. This very important to install and build xfs file system.

Note

It appears to be the fact that xfs, neither officially nor unofficially supported by Redhat. Therefore it not included in the standard SL installation. SLC has it as an external package. So we started rebuilding the rpm from SLC respo.

Post Installation Configuration

Requirements

• kernel-module-xfs
• xfsprogs
• kernel-devel

Installation

rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/45/x86_64/contrib/SRPMS/xfs/kernel-module-xfs-*.src.rpm
rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/45/x86_64/contrib/SRPMS/xfs/xfsprogs-*.src.rpm

Make sure kernel-devel is already installed on the system otherwise the next steps will fail.

export rels=`uname -r`
cd /usr/src/redhat/SPECS
rpmbuild -ba kernel-module-xfs.spec --define "kernel_topdir /lib/modules/${rels}/build"
rpmbuild -ba xfsprogs.spec --define "kernel_topdir /lib/modules/${rels}/build"

It will take some time to complete. And then,

cd /usr/src/redhat/RPMS/x86_64
rpm -ivh kernel-module-xfs*.rpm
rpm -ivh xfsprogs-*.rpm

At this moment xfs is installed; which mkfs.xfs should tell you that. Now we need to prepare the disk array for the file system.

Formatting the Disk Array

As fdisk is not designed for large partitions and doesn’t understand GUID Partition Table (GPT) either so GNU parted was the other alternative. But it's very important to set the label on the disk to gpt to make XFS filesystem to work. parted commands are very simple; say for /dev/sdb, it should be:

parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary xfs 0 ­-0
(parted) quit

Or from the CLI:

export EnD=`parted /dev/sdb p | gawk '{if (NR==1){print $5}}'`
export END=`echo ${EnD} | tr  \\\n | sed ss.*-\\\(digit:\\+\\\)*s\\1s`
parted /dev/sdb p mklabel gpt
parted /dev/sdb mkpart primary xfs 0 ${END}
parted /dev/sdb p

This will actually create a single partition on the entire disk array.

Creating File system

Some point we need create a mount point before we try to mount the file-system. Assuming "/dpm_data" as the mount point, this should create and mount the new xfs file-system.

mkfs.xfs -f /dev/sdb1
mkdir -p /dpm_data

mv /etc/fstab /etc/fstab-ORG
cat /etc/fstab-ORG | sed '/dev\/sdb/d' > /etc/fstab
cat << EOF >>/etc/fstab
/dev/sdb1            /dpm_data               xfs     defaults        1 2
EOF

mount -aF
df -kh

Installation Script

I put together all the commands and made this script to help myself from stop repeating all of those above steps for my other nodes. Maybe not the most clever one but it did pretty well for me on our other disk servers.

#!/bin/bash

#################################################################################
#                                                                               #
#          xfs file-system configuration on 64-bit SL-4.x/SLC-4.x               #
#          Santanu Das, Cavendish Laboratory, 12/07/2007                        #
#          Last Modified : 26/07/2007                                           #
#                                                                               #
#################################################################################

# Color code
ErroR=`echo -e '\E[0;31m'"\033[1m[ ERROR ]\033[0m"`
InfO=`echo -e '\E[0;32m'"\033[1m[ INFO ]\033[0m"`
QueS=`echo -e '\E[0;34m'"\033[1m[ INFO ]\033[0m"`

# Function for exit due to fatal program error
function error_exit()
{
    PROGNAME=$(basename $0)
    echo "${PROGNAME}: ${1:-"Unknown Error"}" 1>&2
    exit 1
}

rpm_dir="/usr/src/redhat/RPMS/x86_64"
ver=`lsb_release -r | cut -f2 | sed 's/\.//g'`
arch=`uname -i`
rels=`uname -r`
smp=`echo ${rels} | tr '' \\\n | sed 's/smp//g'`

function SELECT_DEV()
{
    echo ""
    echo -n "Please enter the name of the Device: "
    read -n 0 DEV
    if [ "${DEV}" == "" ]; then
        DEV_PATH="/dev/$$$"
    else
        DEV_PATH="/dev/${DEV}"
    fi

    if [ ! -e ${DEV_PATH} ]; then
        echo -en "${ErroR} "; echo "Invalid entry: ${DEV_PATH}"
        SELECT_DEV
    else
        echo -en "${InfO} "; echo -n "You have entred: "
        echo -e '\E[0;34m'"\033[1m${DEV_PATH}\033[0m"
        ANS=
        until [ "${ANS} == "E" || "${ANS} == "e" ]; do
             echo ""; printf "Is that correct? [ Y/N | E to exit ]: "
             read -n 1 ANS
             echo ""
             case ${ANS} in
                Y | y ) MKFS ;;
                N | n ) echo -en "${InfO} "; echo "The device is incorrect, as you said...."
                        SELECT_DEV ;;
                E | e ) exit 0 ;;
                * ) echo "";
                    echo -en "${ErroR} "; echo "Wrong entry....."
                    echo -en '\E[0;32m'"\033[1m[Y]\033[0m"; echo "es to proceed"
                    echo -en '\E[0;32m'"\033[1m[N]\033[0m"; echo "o to try again"
                    echo -en '\E[0;32m'"\033[1m[E]\033[0m"; echo "xit to quit" ;;
             esac
        done
fi
}

function MKFS()
{
    # Using parted to create the partition table
    #
    EnD=`parted ${DEV_PATH} p | gawk '{if (NR==1){print $5}}'`
    END=`echo ${EnD} | tr '' \\\\\n | sed ss.*-\\\\\([[:digit:]]\\\\+\\\\\)*s\\\\1s`
    parted -s ${DEV_PATH} rm 1 2 3 4
    parted -s ${DEV_PATH} mklabel gpt
    parted -s ${DEV_PATH} mkpart primary xfs 0 ${END}
    echo ""
    echo -en "${InfO} "; echo "Partitioning complete"
    parted ${DEV_PATH} p

    echo ""
    echo -n "${InfO} "; echo "Creating the file system..........."
    sleep 2; echo "============================================";
    mkfs.xfs -f ${DEV_PATH}1

    echo ""; echo -en '\E[0;32m'"\033[1m"
    echo "=============================================================="
    echo " You need to create a mount point to mount the xfs filesyste."
    echo "      e.g.    /dpm_data"
    echo "=============================================================="
    echo -en "\033[0m"; echo ""
    echo -n "Enter the name of the mount point: "
    read -n 0 MNT_NAME
    mkdir -p ${MNT_NAME}

    # Adding entry to /etc/fstab
    mv /etc/fstab /etc/fstab-ORG
    cat /etc/fstab-ORG | sed '/'${DEV}'/d' > /etc/fstab

cat << EOF >>/etc/fstab
${DEV_PATH}1            ${MNT_NAME}               xfs     defaults        1 2
EOF

    echo ""
    echo -en "${InfO} "; echo "Mounting the file system..........."
    sleep 1; echo "============================================";
    mount -aF
    df -kh

    echo ""
    echo -n "${InfO} "; echo "Installation complete.  Good bye!!!"
    exit;
}

echo ""
echo -n "${InfO} "; echo "Preparing the disks for xfs file system"
echo -n "${InfO} "; echo "Please wait............................"
sleep 2; echo ""

# Install the source rpm
echo -n "${InfO} "; echo "Installing the source rpms............."
rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/${ver}/${arch}/contrib/SRPMS/xfs/kernel-module-xfs-*.src.rpm
rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/${ver}/${arch}/contrib/SRPMS/xfs/xfsprogs-*.src.rpm

echo -n "${InfO} "; echo "Rebuilding the kernel in progress"
echo -n "${InfO} "; echo "It may take a while...................."
sleep 2; echo ""

# Checking for the kernel-devel installation
KER=`rpm -qa | grep kernel-devel-${smp}`
if [ "${KER}" == "" ]; then
        echo -n "${InfO} "; echo "kernel-devel not found; installing....."
        yum install -y kernel-devel-${smp}
fi

cd /usr/src/redhat/SPECS
rpmbuild -ba kernel-module-xfs.spec --define "kernel_topdir /lib/modules/${rels}/build"
rpmbuild -ba xfsprogs.spec --define "kernel_topdir /lib/modules/${rels}/build"
#
# Installing [ Not sure if really required ]
echo -n "${InfO} "; echo "Installing the xfs kernel module......."
rpm -ivh ${rpm_dir}/kernel-module-xfs*.rpm
rpm -ivh ${rpm_dir}/xfsprogs-*.rpm
echo ""
echo -n "${InfO} "; echo -n "xfs is installed in: " 
echo -e '\E[0;34m'"\033[1m`which mkfs.xfs`\033[0m"
#
echo -n "${InfO} "; echo "Formatting the partitioning............."
echo ""; echo -en '\E[0;32m'"\033[1m"
echo "=============================================================="
echo " You need to specify the device you want to format"
echo " Just the name of the device, *NOT* the full path"
echo -n "       e.g. "; echo -en '\E[0;34m'"\033[1msdb\033[0m"
echo -en '\E[0;32m'"\033[1m"; echo " for /dev/sdb"
echo " Be warned that all data will be lost!!!"
echo "=============================================================="
echo -en "\033[0m"
# 
SELECT_DEV