darkness

Today a friend running Red Hat 9 wanted to move two out of three members of a RAID 1 and RAID 5 array from IDE to SCSI drives. (Really he was just making IDE drives look like SCSI drives because he installed the manufacturer’s drivers. Apparently RH kernels’ Highpoint IDE drivers suck.) This is a bit of a problem because the RAID superblock has a list of (major,minor) pairs that make up the devices in the array.

To fix this, we booted up to the RH9 rescue disk (it was a RH9 system) with the Highpoint drivers installed. So now instead of the old members hde and hdg, we had sda and sdb. Nothing started, no /mnt/sysimage because md0 is /boot and md1 is /. raidstart /dev/md0, which is the RAID 1 array, and it starts in degraded mode. Then we used mdadm to add in the missing members of the RAID 1 array. Something like mdadm /dev/md0 -a /dev/sda1 -a /dev/sda2. (sda2 was originally the spare. The kernel raid drivers made it a spare again when we added it.)

I was a bit more worried about what would happen if we tried to bring up the RAID 5 array one disk at a time. So to be safe I used the command from the examples given in the mdadm(8) manual page. mdadm -Ac partitions -m 1 /dev/md1 means “read /proc/partitions, scan every partition, find me partitions that claim to be part of RAID minor number 1 (a.k.a. md1) and add them the device /dev/md1.” For some reason this felt more atomic and thus safer to me. Surprisingly it worked and md1 was now on-line.

Apparently this is all it takes to re-write the RAID superblocks on all involved partitions. You can examine them with mdadm -E /dev/memberdevice (like mdadm -E /dev/sda1) and see what devices are listed as part of the RAID array in the superblock. Don’t forget to reflect your changes in /etc/raidtab.

In other news, mkinitrd adds SCSI support and SCSI modules based on finding alias scsi_hostadapter lines in /etc/modules.conf. To get mkinitrd to automatically add in scsi_mod, sd_mod, and hpt37x2 (the Highpoint driver in this particular case) we just had to alias scsi_hostadapater hpt37x2 in /etc/modules.conf. Also, mkinitrd won’t add software RAID modules unless /proc is mounted and (presumably) showing RAID devices in use. So if you’re chrooted into /mnt/sysimage in RH rescue mode, go ahead and mount /proc inside the chroot before you run mkinitrd -v.

Working with PIKT. I got around to the keys.conf documentation. Looked pretty neat, I decide to tell PIKT to distribute PIKT.conf. (BTW, don’t get too distracted and read keys.cfg instead of keys.conf. I named the file keys.cfg and got kind of confused for a while because PIKT.conf was showing things like private_key CHANGE_ME.)

[root@piktmaster /usr/local/pikt/lib/configs]# piktc -pv +H piktmaster
processing piktmaster.mydomain.net...
killing daemon (piktc_svc)...
root@piktmaster.mydomain.net's password:
installing PIKT.conf...
root@piktmaster.mydomain.net's password:
PIKT.conf                                     100%  186   188.6KB/s   00:00
PIKT.conf installed
restarting daemon (piktc_svc)...
root@piktmaster.mydomain.net's password:
determining piktc_svc pid...
Segmentation fault (core dumped)

This led me digging with GDB, looking at the output of rpcgen, and looking at PIKT sources. Finally I determined two things. First, the problem is the lack of register_request() being called in src/piktc/rpcclnt.c in the install_piktconf() (IIRC) function before piktc_svc_pid() is called. I put some register_request() around it and no more SIGSEGV. Second, this SIGSEGV was kind of unimportant. At this point PIKT has already done as much as it’s going to do. All it’s trying to do at that point, AFAIK, is make sure that piktc_svc restarted by getting its PID. So maybe I could have ignored it, though it might have mucked up distributing to multiple hosts. Hopefully I’ll remember to make and send a patch to the PIKT team/author tomorrow.

Of course, now that I’ve got all this working, I notice that PIKT is leaving ssh processes around after distributing PIKT.conf. Wonder if I’ll fix that or not…

Oh, and I’ve noticed that PIKT is picking different ports to listen on after every invocation. Joy! I’ll have to fix that so I can sic various packet filters on it. Sometimes I wonder if I shouldn’t be running IPSec internally.

Then I realize that it’s easily 1.5h past when I wanted to go to bed, so I shut up and go do that.

I woke up this morning and found I could not get xscreensaver to prompt me for my password. I couldn’t log in as root. I popped up the temporary head I have on my OpenBSD box, fired up tcpdump, and saw nothing of merit. I also forgot to enable Alt-SysRq on my desktop Linux box. So I hit the reset switch. As it’s coming back up, automount (of all things) hangs for a long time. A few reboots and single user modes later, I find there’s another configuration file, /etc/openldap/ldap.conf that contains the LDAP server’s address. In this case I’d changed only /etc/ldap.conf and not /etc/openldap/ldap.conf. I don’t know how it’s determined which file is used, but once I change the LDAP server in /etc/openldap/ldap.conf, everything started working again. If you can, I guess I’d advise just changing it with authconfig in Red Hat, probably. I’m afraid to run it because I suspect it will overwrite some of my custom settings. Sigh.

I’m putting PIKT onto my OpenBSD machine, as a master for my other Linux machines. I’ll probably have PIKT jobs running on the master as well.

The install works fine if you just ./configure && gmake && gmake install. I stuck the sources in /usr/src. If you don’t give --prefix to configure it defaults to sticking everything under a “virtual root” in /usr/local/pikt. What I mean by “virtual root” is that you get /usr/local/pikt/bin where all the PIKT binaries are, /usr/local/pikt/var/log where logs are, etc. Note that I used gmake; install the gmake (GNU make) package. I also had to install the bison package. flex appears to have been on the system. rx is apparently an (old?) GNU regular expression library. I didn’t have to install this — and there wasn’t a package for it. I’ll try and keep the fact that I might be missing a requirement in mind if stuff starts not working as I move forward. (I don’t have gawk either. I don’t know if that’s a requirement or not.)

I’m going through the PIKT “Tutorial and Getting Started”. First note: make sure you don’t have 127.0.0.1 myhostname myhostname.mydomain in /etc/hosts (where myhostname and mydomain are your real host name and domain name) because it mucks things up. Over the years I’ve had various Linux distributions tell me that your host name shouldn’t be in there as 127.0.0.1, and various others put it in there, so I’m not real sure which is correct. However, going through the examples in the above mentioned tutorial with your host name pointed to 127.0.0.1 will cause weird things like call-backs to the master to fail. I commented mine out.

Second note: apparently it’s not safe to run piktc with a relative path. I was sitting in /usr/local/pikt/lib/configs and I tried to execute the first example’s alert with ../../bin/piktc. Lo, piktc_svc somehow decided it would be smart to try and run ../../bin/pikt to send the e-mail relating to my alert? I have no idea how this happens, but when I started piktc with an absolute path (/usr/local/pikt/bin/piktc) it works. Bizarre.

First, let me say, for Andy: this & that. (Why am I saying that again?)

Anyway, on to the point. I decided I would try and set up authentication with NTP on my LAN. I find they have the traditional keys setup, which is documented reasonably well, particularly in the NTP FAQ. Then I read about “autokey”: public key cryptography for NTP authentication. Sounds neat, sound recent, decide to try it.

If you want to take time coding a feature, and you want other people to use it, you might want to document it better than autokey. The main point of autokey, authentication, didn’t seem to be addressed to me, at least not in examples. I feel that I actually read a decent amount of documentation, but all I found were explanations of how the protocol works, scant examples, and people on Usenet basically repeating what I’m saying here. I have no idea, for example, if I’m supposed to copy public keys from my server to my client so they can authenticate. (Some random post to a mailing list may help in showing a configuration example for autokey.) For that matter, I’m not even sure which file that ntp-keygen makes contains the public key (the certificate file?).

Then I realized that I’m not authenticating with the public time servers my local NTP server synchronizes with in the first place. What’s more likely: an attack from the Internet, or an attack from my LAN? For that matter, other than screwing up log files and making cron jobs run, what good is changing my time? You can annoy me, but an attack based on this might be non-trivial. Your best bet might be attacking Kerberos, I guess. To screw me from the Internet you’re going to have to imitate the four NTP servers I synchronize off of anyway. So thppt on authentication.

I realize that ntpd is free software, and maybe I shouldn’t be bitching. I really don’t want this to sound like a bitch, more like: if you want people to use the useful functionality you’ve taken the time to code in, you might want to take some time and give better documentation on it as well. Otherwise all your work may be in vain. I suspect the creators of ntpd and other NTP gurus understand all this, though, so they don’t much care. They did it for themselves and that’s fine and great. Maybe if I had more time — and cared more – I’d write the documentation myself. I’m not pissed at the ntpd creators, though; just kind of temporarily frustrated.

Anyway, syslog doesn’t seem to have any capability for doing things like remote logging over SSL, in either Linux or OpenBSD. Third party packages needed I suppose. Maybe I can stunnel it. Maybe I won’t bother for now.