Rescue: Linux and the Dell DRAC
The Planet took our dedicated server offline for some power maintenance. When I woke up the next morning, the server wasn’t up.
The server has a Dell DRAC III card in it, which allows me to remotely view the POST, get into BIOS settings, and generally interact with anything on the console as long as it’s text based. To use it you connect into the web server on the card (it has its own IP), some Java stuff runs, and you’re eventually presented with a VNC-ish window that displays the console and lets you type just as if you were sitting at the keyboard. I think the current belief is that the card probably snoops on the video memory to relay what is on the console to you, and somehow interacts with the PS/2 keyboard as well.
So when I went to view boot up on the server through the DRAC card to see if I could tell what was wrong… nothing. It went through, saw the drives, powered up fine, and then… just stopped. Unexciting. The hosting company’s technicians did a little poking to no avail, then booted me into Knoppix with sshd for me. I sshed in, was able to mount the SCSI disks (LVM on software RAID 1), so I re-ran grub. Had them eject the Knoppix CD, rebooted, and nothing. They were going to charge me for any further work, including booting Knoppix again on the machine. I was pretty sure it was a Grub problem still, but I needed to get at the disks to see if that was the case.
The DRAC III also has a “remote floppy” feature. It works like this: through the DRAC’s web interface, you tell it where it can TFTP an image from. The floppy image you then sent it appears to the system as if it were… well, the floppy drive. You can boot right off of it, multiple times or just once, with the “virtual floppy” being read-only or read/write.
Now, I’ve booted Linux off of it before. I switched the system from JBOD to software RAID 1/LVM with the help of the remote floppy feature. I’ll be damned if it didn’t take me a pretty long time to figure it all out yesterday, though, so I’ll try and go through the process in a fair amount of detail right here.
The general idea is something like:
- Make a boot floppy with a kernel and LILO, just enough to…
- Mount an NFS root with recovery tools, then…
- Mount the disks and see if I can fix what’s wrong.
First of all, I had to remember how to make a boot disk. This is
slightly embarrassing, but I’ll press on. The DRAC seems to take
whatever you throw at it. I think I threw something like a 6MB image
its way at one point, and the DRAC took the download, didn’t complain,
and seemed to work. The system wouldn’t boot, though. I was probably
trying it with SYSLINUX at that point. My first guess is that I just
didn’t have the geometry right, or SYSLINUX couldn’t handle such an
odd looking floppy disk. However, later I tried to upload a 2.88MB
image to it, set LILO specifically for 80 cylinders, 2 heads, 36
sectors/cylinder, and it still didn’t work. So it might be that the
DRAC just won’t work with anything other than 1.44MB images. So dd
if=/dev/zero of=boot.img bs=1k count=1440. Slap ext2 on boot.img
and loop mount it.
Now you need a kernel to put on the disk. I took down the latest
stable, 2.6.18.something. I stripped it down to as bare minimum as I
could; you need bzImage to fit on a 1.44MB floppy, with room for a
few other files. make allnoconfig is a good place to start, in my
opinion. Among other things, I made sure to compile in my Ethernet
driver (e1000), boot-time network configuration (I think? I didn’t
compile in the DHCP/BOOTP/RARP autoconfiguration because I could use
them, but I believe the parent option of those is for configuring the
network from the kernel command line), my SCSI card’s driver (MPT
Fusion) along with support for SCSI disks, md RAID 1, device mapper
(which I think is needed by LVM), ext2, NFS, and NFS root support. I
got my kernel built down to 1.35MB. In retrospect, I could have built
several things as modules, specifically all the stuff for SCSI, RAID,
and LVM. Once you’ve got the NFS root mounted, you can load the
modules from there.
So once you’ve got bzImage, copy it onto boot.img. Next: LILO. I
did a yum install lilo on the CentOS 4 box that was to be my TFTP
server, and I got something like LILO 21.4. Later I went to
http://lilo.go.dyndns.org/ and saw:
NOTE to RedHat and Fedora users: Do not tell me that you are using “version 21.4-4”. It is now 6 years OUT OF DATE. First try the current release of LILO to see if your problem has been fixed.
Oops. So I downloaded the 22.7.3 binary. This package contained no
boot.b. I strongly suspect this version of LILO has boot.b built
into the lilo binary, or otherwise doesn’t need it. Here’s the
lilo.conf I ended up using:
boot = /dev/loop0
disk = /dev/loop0
bios = 0x00
sectors = 18
heads = 2
cylinders = 80
map = /lilo.map
backup = /dev/null
timeout = 600
prompt
compact
read-write
image = bzImage
label = linux-nfs
root = /dev/nfs
append = "i8042.dumbkbd=1 nfsroot=/srv/storage/storage1/tmp/tp1,nfsvers=3,hard,intr,tcp ip=1.2.3.4:5.6.7.8:1.2.3.1:255.255.255.0"
Explaining starting at the top: if you run lilo with a boot device
of something like /dev/loop0, you get an error such as:
Fatal: Sorry, don't know how to handle device 0x0700
So the trick is to tell it that /dev/loop0 is disk 0 as far as
interrupt 13 is concerned. I don’t know that telling it the geometry
was terribly important, but I was running LILO with the -g option,
which told it to use C/H/S addresses rather than linear or LBA32.
I thought /dev/nfs was interesting: this is a character device
with major 0, minor 255, and you have to create it on the boot disk
image (though you can delete it after you run lilo on the disk, I
didn’t). That bit was taken from Documentation/nfsroot.txt in the
kernel sources, which also explains most of the kernel options there.
i8042.dumbkbd=1 is necessary on the
DRAC as I have
explained before. nfsroot tells the kernel first what path to mount
(the server IP is given later), then NFS options. Despite
nfsroot.txt seeming to say that any option you can give when
mounting an NFS mount point normally can be given here, I found at
least one that didn’t (mountvers=3; rw is not accepted either,
though I don’t think that’s a mount option specific to NFS). Further,
when the kernel sees an option it doesn’t recognize, it stops parsing.
So make sure it doesn’t bitch about any of your NFS options. (In my
case, I had something like ...rw,tcp. I always ended up with UDP,
until I realized I needed to take rw out.) Note that I’ve forced NFS
over TCP, since I’m going to be doing NFS over the Internet. I also
left rsize/wsize to 1024 because I’m not terribly concerned about
performance.
The ip option configures TCP/IP. The colon-separated options I’ve
given there are: this host’s IP, NFS server’s IP, this host’s gateway
IP, this host’s netmask. There are a few other parameters, including
a way to specify the device to be used; see nfsroot.txt again. So
in the above example, 1.2.3.4 is my IP, 5.6.7.8 is the NFS
server’s IP, 1.2.3.1 is my gateway, and 255.255.255.0 is my
netmask. The kernel can get an IP via other methods that I’ve
mentioned above, such as DHCP, but The Planet doesn’t offer those (as
far as I know) so I had to manually configure the interface.
So after I put lilo.conf on the boot disk, I needed a few devices.
Make /dev/nfs as above. I think you’ll also need /dev/loop0 and
/dev/null on your boot disk; I copied them from the system’s /dev.
When you’ve got all that done, I invoked LILO like lilo -v -g -r /mnt
-\C /lilo.conf Note that -r tells LILO to chroot into /mnt, so the
path given for the configuration file (-C) has to be relative to the
boot disk as root. Note that you need to do this while boot.img is
bound to /dev/loop0, either because you did it manually with
losetup or because you used mount -o loop (and the first available
loop device was loop0). My LILO reported no errors; hopefully yours
won’t either. If it does, you may want to look into them.
Next I went ahead and set up my NFS root. The tool I always reach for
when I want a minimal, tiny set of the usual binaries is the wonderful
BusyBox: everything you need in a single
binary, including a shell, vi, modprobe, and even a udev
implementation now. So build BusyBox with the ability to run busybox
-\-install when you chroot into your root, and let it install all
sorts of wonderful utilities you need. You’ll probably want to build
in its ability to be /sbin/init, obviously a shell (I like ash),
and build in the aforementioned udev implementation, mdev. mkdir
/dev on the NFS root and copy in (at least), /dev/console,
/dev/tty[01], and /dev/null. I don’t know if you’ll need more or
less than this; I copied in /dev/tty[0-6], /dev/fd0, and
/dev/zero as well. mkdir /proc /sys. mkdir -p /etc/init.d and
put the following in as rcS:
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mdev -s
echo /sbin/mdev >/proc/sys/kernel/hotplug
BusyBox’s init will run this script, which will mount /proc, mount
/sys, and set up its udev implementation. Using mdev
automatically gave me useful device nodes that the kernel knew about,
such as /dev/sda and /dev/md0. I think it might also create the
nodes for the LVM devices when we bring them up.
Speaking of which, you’ll need the LVM utilities. CentOS (and RHL,
and presumably RHEL, and FC) include /sbin/lvm.static, which I think
is used by their initrd. I just copied this over into /sbin on the
NFS root.
I also made a couple convenience scripts in the NFS root for mounting
and unmounting my SCSI disks. Here’s mountall:
#!/bin/sh
lvm.static vgchange -a y
mount /dev/vgmain/lvroot /mnt
mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount /dev/vgmain/lvusr /mnt/usr
mount /dev/vgmain/lvvar /mnt/var
mount /dev/md0 /mnt/boot
I was actually hitting the DRAC console from IE running on XP running
inside VMware on my Linux box. The console is a little laggy, so the
less typing I had to do the better. The above script activates my LVM
devices (and like I said, I don’t know if LVM makes the nodes like
/dev/vgmain/lvroot or if that’s BusyBox’s mdev) and then mounts my
SCSI disks under /mnt.
So we’ve got a disk to boot off of, and an NFS root for that boot disk
to mount. There’s one final piece of the puzzle: the TFTP server.
This isn’t typically a problem… except in talking to the DRAC card
at The Planet, where their firewall apparently breaks the TFTP server
that comes with CentOS (and probably many other distributions,
including probably everything RH makes). The system’s in.tftpd gets
the TFTP request just fine, but then it responds to the request from a
random source port; that doesn’t make it through The Planet’s
firewalls. So I wrote a minimal TFTP server in
Python that
sends the requested file back on the tftp port. This server only
serves a single file (its first command line argument) no matter what
is requested, and only implements TFTP get requests, but it definitely
works to upload the boot image to the DRAC.
So once you’ve got all this set up (TFTP server open to the DRAC card,
NFS server open to the server in distress, all the files and devices
in place…) you’re ready to go. On the DRAC card, tell it to TFTP
your boot.img from tptftpserver.py. It will take a while, then
load it. Tell it to use the image every boot, not just once; this
will prevent you from having to upload the image multiple times (which
gets very annoying very fast). Then you can tell the DRAC to power
cycle your server. If you don’t see the LILO menu/prompt, make sure
that the BIOS knows to try booting off of the floppy drive (“A:” as
this Dell server’s BIOS put it) first. Let LILO boot, hopefully it’ll
hit your NFS server. BusyBox’s init will run, set up lots of stuff
in /dev, then tell you to hit enter to get a shell on the console.
At this point, I mounted up my drive and poked around at Grub more.
The problem, I found, were apparently corrupt or outdated *stage*
files in /boot/grub; they did not match what was in
/usr/share/grub/i386-redhat. grub-install usually copies them
over from /usr/share/grub/i386-redhat to /boot/grub, but
grub-install in RHEL 4 doesn’t work on software RAID /boot (I
think but I’m not positive) so I never run it, and this was a box
that was upgraded using yum from RHEL 3 to CentOS 4. Anyway, copying
the files manually into /boot/grub, then reinstalling Grub (on both
disks! Don’t forget you could be booting off of sdb one day) fixed
my server.