Adding XFS support to 64-bit SL/SL4.x for DPM Disk Server
Contents
Hardware / OS
- System :: Dual-Core AMD Opteron 2210, 4GB of memory
- RAID :: 3Ware/AMCC 9650SE-16M [RAID6]
- Storage :: ~7.5TB per unit (usable), configured as 13-disk RAID6 plus hot spare.
- (Could be 14-disk since RAID6, but we prefered the hot spare configuration)
- OS :: SL-4.5 [ x86_64 ]
- Installation :: Kickstart and manual combined
Problem-1
RHEL4, update 5 only provides the in-built support for 3Ware 9650SE RAID controller. So any version below 4.5 (SL or SLC) simply won't work out of the box. There are some drivers available for selected lower version of SL/SLC on the 3Ware/AMCC website. We started with SL4.5 just make the installation as simple as possible.
Problem-2
It would appear that fdisk has a 2TB partition limit and the partition table labeled as msdos or BSD also doesn't work for a partition larger that 2TB, hence post-installation boot eventually fails due a misconfigured “fstab” file. As we wanted to have a single ~7.5TB partition on each unit for DPM file system, we initially excluded the disk array from the kickstart configuration file and formated afterwards the kickstart installation.
OS Installation
We started with normal kickstart installation but excluding /dev/sdb (which is actually the 7.5TB disk array) and with the following packages.
# Package install information %packages --resolvedeps @ Editors @ Text-based Internet @ Development Tools @ Administration Tools @ System Tools @ yum kernel kernel-smp kernel-devel kernel-smp-devel e2fsprogs kernel-smp-devel grub
Just make sure to install kernel-devel/kernel-smp-devel. This very important to install and build xfs file system.
Note
It appears to be the fact that xfs, neither officially nor unofficially supported by Redhat. Therefore it not included in the standard SL installation. SLC has it as an external package. So we started rebuilding the rpm from SLC respo.
Post Installation Configuration
Requirements
- kernel-module-xfs
- xfsprogs
- kernel-devel
Installation
rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/45/x86_64/contrib/SRPMS/xfs/kernel-module-xfs-*.src.rpm rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/45/x86_64/contrib/SRPMS/xfs/xfsprogs-*.src.rpm
Make sure kernel-devel is already installed on the system otherwise the next steps will fail.
export rels=`uname -r` cd /usr/src/redhat/SPECS rpmbuild -ba kernel-module-xfs.spec --define "kernel_topdir /lib/modules/${rels}/build" rpmbuild -ba xfsprogs.spec --define "kernel_topdir /lib/modules/${rels}/build"
It will take some time to complete. And then,
cd /usr/src/redhat/RPMS/x86_64 rpm -ivh kernel-module-xfs*.rpm rpm -ivh xfsprogs-*.rpm
At this moment xfs is installed; which mkfs.xfs should tell you that. Now we need to prepare the disk array for the file system.
Formatting the Disk Array
As fdisk is not designed for large partitions and doesn’t understand GUID Partition Table (GPT) either so GNU parted was the other alternative. But it's very important to set the label on the disk to gpt to make XFS filesystem to work. parted commands are very simple; say for /dev/sdb, it should be:
parted /dev/sdb (parted) mklabel gpt (parted) mkpart primary xfs 0 -0 (parted) quit
Or from the CLI:
export EnD=`parted /dev/sdb p | gawk '{if (NR==1){print $5}}'` export END=`echo ${EnD} | tr \\\n | sed ss.*-\\\(digit:\\+\\\)*s\\1s` parted /dev/sdb p mklabel gpt parted /dev/sdb mkpart primary xfs 0 ${END} parted /dev/sdb p
This will actually create a single partition on the entire disk array.
Creating File system
Some point we need create a mount point before we try to mount the file-system. Assuming "/dpm_data" as the mount point, this should create and mount the new xfs file-system.
mkfs.xfs -f /dev/sdb1 mkdir -p /dpm_data
mv /etc/fstab /etc/fstab-ORG cat /etc/fstab-ORG | sed '/dev\/sdb/d' > /etc/fstab cat << EOF >>/etc/fstab /dev/sdb1 /dpm_data xfs defaults 1 2 EOF
mount -aF df -kh
Installation Script
I put together all the commands and made this script to help myself from stop repeating all of those above steps for my other nodes. Maybe not the most clever one but it did pretty well for me on our other disk servers.
#!/bin/bash ################################################################################# # # # xfs file-system configuration on 64-bit SL-4.x/SLC-4.x # # Santanu Das, Cavendish Laboratory, 12/07/2007 # # Last Modified : 26/07/2007 # # # ################################################################################# # Color code ErroR=`echo -e '\E[0;31m'"\033[1m[ ERROR ]\033[0m"` InfO=`echo -e '\E[0;32m'"\033[1m[ INFO ]\033[0m"` QueS=`echo -e '\E[0;34m'"\033[1m[ INFO ]\033[0m"` # Function for exit due to fatal program error function error_exit() { PROGNAME=$(basename $0) echo "${PROGNAME}: ${1:-"Unknown Error"}" 1>&2 exit 1 } rpm_dir="/usr/src/redhat/RPMS/x86_64" ver=`lsb_release -r | cut -f2 | sed 's/\.//g'` arch=`uname -i` rels=`uname -r` smp=`echo ${rels} | tr '' \\\n | sed 's/smp//g'` function SELECT_DEV() { echo "" echo -n "Please enter the name of the Device: " read -n 0 DEV if [ "${DEV}" == "" ]; then DEV_PATH="/dev/$$$" else DEV_PATH="/dev/${DEV}" fi if [ ! -e ${DEV_PATH} ]; then echo -en "${ErroR} "; echo "Invalid entry: ${DEV_PATH}" SELECT_DEV else echo -en "${InfO} "; echo -n "You have entred: " echo -e '\E[0;34m'"\033[1m${DEV_PATH}\033[0m" ANS= until [ "${ANS} == "E" || "${ANS} == "e" ]; do echo ""; printf "Is that correct? [ Y/N | E to exit ]: " read -n 1 ANS echo "" case ${ANS} in Y | y ) MKFS ;; N | n ) echo -en "${InfO} "; echo "The device is incorrect, as you said...." SELECT_DEV ;; E | e ) exit 0 ;; * ) echo ""; echo -en "${ErroR} "; echo "Wrong entry....." echo -en '\E[0;32m'"\033[1m[Y]\033[0m"; echo "es to proceed" echo -en '\E[0;32m'"\033[1m[N]\033[0m"; echo "o to try again" echo -en '\E[0;32m'"\033[1m[E]\033[0m"; echo "xit to quit" ;; esac done fi } function MKFS() { # Using parted to create the partition table # EnD=`parted ${DEV_PATH} p | gawk '{if (NR==1){print $5}}'` END=`echo ${EnD} | tr '' \\\\\n | sed ss.*-\\\\\([[:digit:]]\\\\+\\\\\)*s\\\\1s` parted -s ${DEV_PATH} rm 1 2 3 4 parted -s ${DEV_PATH} mklabel gpt parted -s ${DEV_PATH} mkpart primary xfs 0 ${END} echo "" echo -en "${InfO} "; echo "Partitioning complete" parted ${DEV_PATH} p echo "" echo -n "${InfO} "; echo "Creating the file system..........." sleep 2; echo "============================================"; mkfs.xfs -f ${DEV_PATH}1 echo ""; echo -en '\E[0;32m'"\033[1m" echo "==============================================================" echo " You need to create a mount point to mount the xfs filesyste." echo " e.g. /dpm_data" echo "==============================================================" echo -en "\033[0m"; echo "" echo -n "Enter the name of the mount point: " read -n 0 MNT_NAME mkdir -p ${MNT_NAME} # Adding entry to /etc/fstab mv /etc/fstab /etc/fstab-ORG cat /etc/fstab-ORG | sed '/'${DEV}'/d' > /etc/fstab cat << EOF >>/etc/fstab ${DEV_PATH}1 ${MNT_NAME} xfs defaults 1 2 EOF echo "" echo -en "${InfO} "; echo "Mounting the file system..........." sleep 1; echo "============================================"; mount -aF df -kh echo "" echo -n "${InfO} "; echo "Installation complete. Good bye!!!" exit; } echo "" echo -n "${InfO} "; echo "Preparing the disks for xfs file system" echo -n "${InfO} "; echo "Please wait............................" sleep 2; echo "" # Install the source rpm echo -n "${InfO} "; echo "Installing the source rpms............." rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/${ver}/${arch}/contrib/SRPMS/xfs/kernel-module-xfs-*.src.rpm rpm -ivh ftp://ftp.scientificlinux.org/linux/scientific/${ver}/${arch}/contrib/SRPMS/xfs/xfsprogs-*.src.rpm echo -n "${InfO} "; echo "Rebuilding the kernel in progress" echo -n "${InfO} "; echo "It may take a while...................." sleep 2; echo "" # Checking for the kernel-devel installation KER=`rpm -qa | grep kernel-devel-${smp}` if [ "${KER}" == "" ]; then echo -n "${InfO} "; echo "kernel-devel not found; installing....." yum install -y kernel-devel-${smp} fi cd /usr/src/redhat/SPECS rpmbuild -ba kernel-module-xfs.spec --define "kernel_topdir /lib/modules/${rels}/build" rpmbuild -ba xfsprogs.spec --define "kernel_topdir /lib/modules/${rels}/build" # # Installing [ Not sure if really required ] echo -n "${InfO} "; echo "Installing the xfs kernel module......." rpm -ivh ${rpm_dir}/kernel-module-xfs*.rpm rpm -ivh ${rpm_dir}/xfsprogs-*.rpm echo "" echo -n "${InfO} "; echo -n "xfs is installed in: " echo -e '\E[0;34m'"\033[1m`which mkfs.xfs`\033[0m" # echo -n "${InfO} "; echo "Formatting the partitioning............." echo ""; echo -en '\E[0;32m'"\033[1m" echo "==============================================================" echo " You need to specify the device you want to format" echo " Just the name of the device, *NOT* the full path" echo -n " e.g. "; echo -en '\E[0;34m'"\033[1msdb\033[0m" echo -en '\E[0;32m'"\033[1m"; echo " for /dev/sdb" echo " Be warned that all data will be lost!!!" echo "==============================================================" echo -en "\033[0m" # SELECT_DEV