darkness

We just got another server at The Planet. They had a deal on another of the same 1600SC’s we got before: dual Xeon, but this time with dual 10k 136GB SCSI disks.

They offer CentOS 5 now. (When we bought our last one, they only offered RHEL 3, which I had to remotely upgrade to CentOS 4.) When they turn the server over to you, you’ve got /, /boot, swap, and then /dev/sdb is one big partition. My task: change it to LVM on software RAID 1.

I don’t think their web site mentions it, and there was some question about whether they still offer it, but this newest server still has a Dell DRAC III card in it, which I needed to toy with grub (when I screwed up). You could probably do without remote console access, except (1) it will be hard to see errors, and (2) I don’t recall if grub has a functionality similar to lilo -R (“for the next boot only, use boot entry N”) these days.

Here are my rough steps to complete this task. This is based partly on what I did, partly what I should have done, and it’s all from memory (granted, memories that are scant hours old). Note that you probably don’t want to do this on a “production” machine (duh), as this will require reboots and potentially horking things. I did it on a fresh install that I wasn’t afraid to have reinstalled; you’ve been warned.

1. Partition /dev/sdb. For me, that’s 128MB /dev/sda1 as /boot, and the rest in /dev/sdb2. Both types 0xfd for RAID auto-detect. I set sdb1, which will be software RAID 1 for /boot, to be an “active” (AFAIK means “can boot off of here”) partition, in case sda1 fails.
2. Create the software RAID devices. For /boot, this like like mdadm --create -l 1 -n 2 /dev/md0 missing /dev/sdb1. missing is the word I always forget: it says “the disk that would normally go in this slot is… well, missing.” Repeat for /dev/md1 with /dev/sdb2.
3. pvinit, vgcreate, and then lvcreate all the logical volumes you want.
4. mkfs or equivalent on all your new devices. For the record, I found out RH’s anaconda does something like mke2fs -j -i <inode size> <device>. I left out the -i. Other options for mke2fs are apparently read from /etc/mke2fs.conf. After mke2fs anaconda seems to do tune2fs -i0 -c0 -ouser_xattr,acl -Odir_index <device>. (This is from memory, I hope it’s right.) Don’t forget to e2label everything too. RHEL 5’s mkswap apparently takes -L to set a label, which you can use in /etc/fstab.
5. Mount up your new, empty filesystem(s) somewhere. I made and used /mnt/sdb.

6. Edit grub.conf. Note on my system, /etc/grub.conf was not symlinked to the actual file, /boot/grub/grub.conf. If you see this I don’t see why you shouldn’t rm /etc/grub.conf && ln -s ../boot/grub/grub.conf /etc/grub.conf like I did.

   In grub.conf you want to make a copy of your currently booted stanza. In this copy, change the root (hd0,0) to be root (hd1,0), and also change the root=LABEL=/ in the kernel line to point at something like root=/dev/vgmain/lvroot (or wherever your root FS is).

   Also, for people like me on a DRAC III, you’ll want to comment out any bootsplash or hiddenmenu lines, you don’t need any serial console (I commented out both serial and terminal lines for GRUB and also removed console=... parameters to the kernel). You may want to bump up the timeout. Additionally, DRAC III users will need to add i8042.dumbkbd=1 to the kernel “command line.”

7. I don’t think this is necessary… but I did add (hd1) /dev/sdb to /boot/grub/device.map and I also re-ran /sbin/grub-install.

8. Copy everything from the running filesystems to the new filesystem(s). I used cd /; find . -xdev -print | cpio -pvdm /mnt/sdb/ to do this, and then the same with /boot. Jumping ahead some steps, after I finally got the new arrays booted, I don’t think this copied the SELinux labels right, which is no good. I encourage you to find a command that does copy the SELinux labels. Maybe star can help? Or cp -ax? If you use my method, go ahead and touch .autorelabel in the root of your new filesystem, which should cause SELinux relabelling at boot.
9. chroot to your new FS. mount /proc && mount /sys.
10. Edit /etc/fstab on the new FS to reflect your new layout. I note that a fresh CentOS 5 install using LVM on RAID doesn’t use LABEL=... syntax for selecting devices, so I’d put actual device names for everything on software RAID or LVM in column 1 of fstab. Note that I never got mkinitrd to run correctly (i.e. include the LVM stuff) as long as I was using LABEL=/ for my / partition.
11. This is kind of weird: I think you need the device nodes hanging around to ensure that mkinitrd works right in the next step, so run start_udev. The weird part: this is going to kill the running udevd and replace it with one whose / points to your new root. I was able to finish my work and reboot my system without udevd running on my real root; hopefully your mileage won’t vary.
12. If you’re using LVM like me, you need to run vgmknodes now, to make things like /dev/vgmain/lvroot (example name).
13. Now it’s time to run mkinitrd. rm the existing initrd in /boot and run mkinitrd with the -v switch. Make sure it pulls in lvm (and look for lvm.conf too).
14. Theoretically you’re now ready to go, and it’s time to reboot. (Last chance to touch /.autorelabel.) killall udevd; umount sys && umount proc && exit to get out of the chroot. umount the rest of the stuff for fun, maybe vgchange -a n and then mdadm --stop all your new RAID devices (since I’m paranoid and not sure if RHEL would do this for me, since they weren’t there at boot), and reboot.
15. Hopefully you can now select your new GRUB entry at boot, and boot from your new disks. (If you touched .autolabel you’re in for an extra reboot.) Once it boots, log in and make sure mount looks right, with your new LVM/RAID devices. Also check /proc/swaps to make sure the right swap device is being used.
16. Once you’re sure you’re OK, and you don’t want anything off the old disk, sfdisk --dump /dev/sdb | sed 's/sdb/sda/g' | sfdisk /dev/sda to copy the partition table from sdb to sda.
17. mdadm --manage --add the sda partitions to the RAID devices.
18. Wait for resync (i.e. bounce on cat /proc/mdstat).
19. Now go back to grub.conf, take out your extra entry (the one using (hd1,0)). Change the old entry (in my grub.conf I now only had one entry) to use the real root device path, so (ex.) root=/dev/vgmain/lvroot) instead of root=LABEL=/.
20. Make sure /etc/fstab looks right still. Again, I recommend against using LABEL= syntax for anything on software RAID or LVM.
21. I re-ran mkinitrd with -v, more than anything just to make sure it worked right. (Again: if your / is on LVM, make sure you see things like lvm.conf being put on the initrd. I didn’t the first time I did this, and it turned out to be because I was using LABEL=... syntax in /etc/fstab for /.)
22. Optional: I wrote a GRUB boot sector to sdb as if it were the first disk in the system. Run grub, device (hd0) /dev/sdb, root (hd0,1), setup(hd0). You can check that GRUB is now on sdb with something like dd if=/dev/sdb bs=512 count=1 | strings | grep GRUB and you should see GRUB as output.
23. Reboot (and pray, perhaps).

Once you’ve rebooted this last time you’ll probably want to look at mount to make sure everything is still where it should be, check your swap is on, and check /proc/mdstat to make sure the arrays are OK. In my case, and slightly to my surprise, everything was OK.

Sorry to the folks at The Planet if my turning off serial console stuff in GRUB/the kernel screws them up. (Feel free to drop me a line and I’ll turn it back on and update these instructions.)