darkness

[Updated 2002-01-18 02:21 EST: I found out later about the bos adduser comand I forgot. By the time I found out about this in the OpenAFS section "Configuring the Top Levels of the AFS Filespace" I had already stopped running with no authentcation. This meant I couldn't figure out how to issue the command since it wouldn't give me permission to bos adduser. Reading some documentation I found out about the -localauth argument. This argument, apparently accepted by many commands, instructs the command to authenticate you as root. In other words, if you want to use -localauth you have to be root. Anyway, I found I could do a bos adduser verin.caliginous.net afsmaster.admin -cell caliginous.net -localauth as root and be on my way.]

OpenAFS seems very easy to set up. The OpenAFS documentation is quite good and has specific sections for Linux. They have OpenAFS RPMS for Red Hat going back to something like 6.2.

I installed all the RPMS on my first AFS cell, then went to the part of the quick start guide that’s entitled “Installing the First AFS Machine.” However, I was aware that AFS has its own Kerberos IV server, but that I wanted to use my MIT Kerberos V installation instead. The documentation on the OpenAFS site is not helpful here, instructing you to contact IBM for more information. Yeah, right.

Thankfully, someone wrote instructions on installing OpenAFS with another Kerberos implementation. These instructions are pretty good, but they’re a bit sketchy at points. I’ll try to fill in some holes here. If I feel brave maybe I’ll add them to the Wiki page, or at least a link to this log entry. This log entry was written for revision 1.10 of the Wiki page mentioned above.

Starting on the Wiki page at “Starting the Install” I followed all instructions. They don’t explicit state at what step you’re supposed to stop following the OpenAFS documentation and start following their steps. I stopped before I started the section in the OpenAFS documentation entitled “Initializing Cell Security.” You’re going to basically skip this entire step, though the instructions on the Wiki page are the same as the instructions in the OpenAFS documentation in several places.

Once I got to the section in the Wiki page entitled “Create AFS Keys and Administrators,” I created a principle called afs/myrealm.com using kadmin. At first I tried just doing addprinc -randkey afs/mycell.com, but that seemed to potentially be in conflict with the instructions that are given in the Heimdal KTH instructions (which are referred to and somewhat used in the MIT Kerberos V instructions). Here’s what I ended up doing:

[darkness@servername ~]# kadmin -p kmaster/admin
kadmin:  addprinc -randkey -e des-cbc-crc:v4 afs/mycell.com
WARNING: no policy specified for afs/mycell.com@MYREALM.COM;
defaulting to no policy
Principal "afs/mycell.com@MYREALM.COM" created.
kadmin:  getprinc afs/mycell.com
Principal: afs/mycell.com@MYREALM.COM
Expiration date: [never]
Last password change: Fri Jan 17 05:35:43 EST 2003
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Fri Jan 17 05:35:43 EST 2003 (kmaster/admin@MYREALM.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 2, DES cbc mode with CRC-32, no salt
Principal: afs/mycell.com@MYREALM.COM
Expiration date: [never]
Last password change: Fri Jan 17 05:35:43 EST 2003
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Fri Jan 17 05:35:43 EST 2003 (kmaster/admin@MYREALM.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 2, DES cbc mode with CRC-32, no salt
Attributes:
Policy: [none]

I did the getprinc just to see if it worked. You presumably have to add the -e des-cbc-crc:v4 to make it generate only the key type that’ll be understood by AFS. In retrospect I suspect you can leave the -e switch out and use the parameters to ktadd to only export the des-cbc-crc:v4 key; someone is welcome to try this.

Note that there are instructions here about how your Kerberos V realm must respond to Kerberos IV requests. If you followed the RH documentation on setting up your Kerberos server, you’ve got krb524 running and you’re set to go with Kerberos IV requests.

Next the Wiki page instructs you to create the /usr/afs/etc/KeyFile file with the key for the principle we just created, above. This is where I had to feel my way around just a little bit. For your benefit, I’ll quote important parts of my session below. Keep in mind that we’re still on the Wiki page instructions here, having left the OpenAFS documentation when we hit the “Initializing Cell Security” section. Also note that the following starts in the same kadmin session as above that we never exited.

kadmin:  ktadd -k /etc/afskeytabfile.krb5 -e des-cbc-crc:v4 afs/mycell.com
Entry for principal afs/mycell.com with kvno 4, encryption type
DES cbc mode with CRC-32 added to keytab WRFILE:/etc/afskeytabfile.krb5.
kadmin:  quit
[darkness@servername afs-working]$ sudo /usr/kerberos/sbin/ktutil
ktutil:  read_kt /etc/afskeytabfile.krb5
ktutil:  list
slot KVNO Principal
---- ----

Started today by getting a call from a client telling us one of their Internet circuits was having problems. Turned out to be the USLEC circuit. Called them, they said they were having some big routing issue and that it’s pretty much covered. OK, no problem, tell the client that USLEC is aware of it and they’re working on it. Several hours pass and the client wants to know what’s going on. I offer to open a ticket with USLEC. First call: open a ticket, think she pulled the wrong customer address up in her database. Call back in 15m to correct. Call 2: corrected address. Call 3, a good three to four hours later: guy calling me asking why I can’t get my e-mail. “Actually, I said that our address seems to be unreachable from certain providers, like Level 3 and Qwest.” Now, maybe I’m wrong here, but that’s how I interpret it when their routers start throwing back ICMP network unreachable messages to me in a traceroute. I know the circuit isn’t completely down because I can get to it from at least one circuit that goes to TWTC. This guy probably called me at least six times, and it took him the first three to get the right physical address of the client’s site. Then he was trying to insist that, because he couldn’t ping the WAN side IP, LMI must be down. … Yeah. In his last call, he asked me “if I had ‘ip classless’.” When I responded affirmatively, he informed me that they were going to have to escalate. By now this is like 2030. He says the team he’s escalating to doesn’t have anyone there at night (or maybe just this night) so it’ll be until morning. Needless to say, my client is endlessly pleased with USLEC right now.

Anyway, after some squabbling in the morning I was off to working on RT. It was pretty much set up, I just had to learn how to do a few things. Oh, and troubleshoot the e-mail, which was kind of a bitch. Because of the DNS setup we have (explained in past entries) everything we use pretty much gets CNAME’d to a host in one particular zone. Unfortunately, here I had support.foo.com that needed to receive mail via mail.foo.com. If you use a CNAME with a label (host) you can’t use any other data, including an MX. What I ended up having to do was make a duplicate name for the IP in our “master” zone and then add the MX to that, then CNAME to that from outside the master zone. More unfortunateness: when support.foo.com is CNAME’d to support.bar.com, and you send mail to support.foo.com, the mail actually ends up trying to be delivered to support.bar.com. Not a big problem, but I set everything up to look for support.foo.com. Fixed soon enough.

Another problem I thought I was having with RT was that replies to ticket comments weren’t getting sent to the person listed in the Cc field. As it turns out, this is because the address associated with my user was in the Cc field, and it prevent you from sending yourself your own comments in e-mail. I just didn’t see this clearly documented, so I ended up dicking with it for a long time.

Oh ho ho, one more problem I had with RT. Using $MailCommand = 'sendmailpipe' it complained about an insecure $ENV{PATH} because it was running in taint mode. Somehow, though $ENV{PATH} gets set in webmux.pl which is PerlRequire‘d in the Apache configuration for the virtual host, it doesn’t care about this and still thinks it unclean. So I had to explicitly local $ENV{PATH} = '/bin:/usr/bin' in lib/RT/Actions:SendEmail.pm around line 119 IIRC. Additionally, it may be helpful to note that the other options for $MailCommand are documented in the man page for Mail::Mailer. As it turned out, since I was using qmail I could just set $MailCommand = 'qmail' and everything seems to be working out.

In other more exiting news, I’ve fixed my problem with Kerberos and OpenSSH that I described last night! The key is a patch to add Kerberos/GSSAPI support in OpenSSH. As it turns out, in the openssh-3.1p1-6.src.rpm that’s used to build the packages for (I think) Red Hat 7.x, you just need to change the release of the package to end in gss and the patch will be applied. In RH 8.0′s OpenSSH SRPM the patch isn’t included, but you can download it from the above page and put it in place of the patch that is commented out in the spec file. Then just tag gss on the end of the release, rebuild, and there you are. Also, note that apparently the OpenSSH SRPM for RH 7.x needs autoconf 2.53 or so from RH 8.0, yet this package itself apparently requires RH 8.0′s Perl 5.8.0 RPM because of a dependency on perl(find.pl). Rather than totally hose up my system by trying to go to this new version of perl, I opted to --nodeps the install of the two autoconf* RPMs from RH 8.0. It worked for building openssh-3.1p1-6.src.rpm on a RH 7.2 box. If it breaks in the future I’ll try and remember to speak of it.

One more note that’s crucial to make the above work: your KDC doesn’t have a host principal by default. You need to create one, as in the instructions for setting up a Kerberos client in the RH docs. If you don’t have /etc/krb5.keytab, you apparently don’t have a host principal set up and the seamless authentication of Kerberos won’t help you when connecting to that machine. At least, this was the case with OpenSSH.

Finally, for Andy we have a screenshot from darkbook.

I think I’ve got Kerberos pretty well set up. Followed the instructions in the RH reference guide pretty much exactly. Even have a RH 7.2 box that’s able to kinit. I can’t think of any real snags I hit during the process of setting up the client or the server.

One apparent disappointment is that if I kinit, ssh doesn’t magically use this token to authenticate me, apparently. I still have to enter a password. I used authconfig on the Kerberos server (which is also the server I’m trying to SSH to — is this a problem?) to turn off LDAP and turn on Kerberos. (I have a diff like I did against /etc for LDAP, but I have to clean it up. Maybe I’ll post it tomorrow.) It seems, though, that pam_krb5 just bounces your password against the Kerberos server to see if it sticks, somehow. I guess this makes sense in the end: the client has to have the support to pass along a ticket to the service. Another interesting note: the SSH client that ships in RH 7.2, at least, has no mention of the word “Kerberos” in it, or so strings `which ssh` | grep -i kerberos reports at least. Additionally, ssh -o 'KerberosAuthentication yes' doesn’t work. I need to check the OpenSSH sources and see what kind of Kerberos support it has. I guess if it passes some sort of AFS tickets I’ll be happy enough. I found a few things on Google that kind of indicate that, at the very least, I’ll have to apply some patches to OpenSSH to get some Kerberos authentication support — and then I lose PAM authentication support, supposedly. This is entirely unconfirmed. Look, fuck you! I need to sleep some time. I’ll check it out tomorrow perhaps. Though I should really get RT working, probably.

In other news, I installed YAPS on my Palm to keep some passwords in. With these Kerberos passwords that I just made up, I decided I needed somewhere to keep them. YAPS supposedly uses Blowfish, the author supposedly built it for his own needs, and it seems to work well enough. Maybe I should strings the files that Backup Buddy transfers and see if I have any password lying about.

Today I decided that I should find a way to do centralized directory services and shared file systems. I decided that my Windows boxes need to get access to this easily as do my Linux boxes. After some searching around, I suspect I’m going to end up with LDAP+Kerberos V+OpenAFS.

A brief discussion of what I’ve considered and discarded. I feel funny about mounting things via Samba from other Unix boxen. I also don’t know of a good way to have my Samba server acting as, say, a PDC authenticate against, say, an LDAP database. Samba can’t be an AD master server or whatever they’re called. (BTW, apparently MS’ AD LDAP server has some weird shit, like generated values for some keys. Very strange.) NFS is OK, but I don’t think that’d allow me the control I want in Windows.

So far I appear to have directory services in LDAP. Try reading something like the Red Hat manual on authenticating against OpenLDAP. This is pretty much correct. Note that RH8′s /etc/services borks up the migration scripts that come with nss_ldap or whatever. I recommend something like this for running migrate_all_online.sh:

perl -ne 'next if (/^s*#/);s/s+/ /g;s/s*#.*//;$_.="n";print' 
         /etc/services | sort | uniq > /tmp/services
ETC_SERVICES=/tmp/services ./migrate_all_online.sh

Even then I had some problems because there were apparently three echo service entries. I think I just deleted the one for the ddp protocol (what’s DDP again?) and it worked.

To graphically edit stuff I dug up GQ. Don’t use 0.6.0, since it appears to have problems binding correctly. (As in, I couldn’t modify anything when I was supposed to have bound to my rootdn.) I used 0.7.0 beta 2 instead, which I got from GQ’s SourceForge.net project page. You need the GQ language pack 0.7.0beta2-0 or something, too. Note that GQ has a spec file in it; I never managed to make rpmbuild -ta gq*.tar.gz work. Also, to build GQ 0.7.0 beta 2 in RH8, I suggest modifying the %build section to read as:

%build
./configure --with-included-gettext --prefix=%{prefix}
aclocal
automake --add-missing
make

IIRC the aclocal and automake --add-missing above were required to make some depmod script/program/link/whatever that it’ll look for. If you get some error about a missing ./depmod or something like that, try the above. (Can you tell how familiar I am with the auto* tools?)

I used RH’s authconfig to set up LDAP directory services and authentication. It appears to have worked. I did a diff -urN on a version of the directory before and after the authconfig. To see what changed check out my /etc LDAP changes patch.

Next: Kerberos V. Stay tuned.

Warning: DOMDocument::loadHTML() [function.DOMDocument-loadHTML]: Unexpected end tag : p in Entity, line: 9 in /srv/www/darkness.codefu.org/root/wordpress/wp-content/plugins/postxform/postxform.php on line 43

Warning: DOMDocument::loadHTML() [function.DOMDocument-loadHTML]: Unexpected end tag : p in Entity, line: 12 in /srv/www/darkness.codefu.org/root/wordpress/wp-content/plugins/postxform/postxform.php on line 43

Warning: DOMDocument::loadHTML() [function.DOMDocument-loadHTML]: Unexpected end tag : p in Entity, line: 16 in /srv/www/darkness.codefu.org/root/wordpress/wp-content/plugins/postxform/postxform.php on line 43

Warning: DOMDocument::loadHTML() [function.DOMDocument-loadHTML]: Unexpected end tag : p in Entity, line: 30 in /srv/www/darkness.codefu.org/root/wordpress/wp-content/plugins/postxform/postxform.php on line 43

Tonight is quick tips night on darky’s web log.

The differences between the Asus A7M-266 and A7V-266-E seem to be:

• A7M has AMD chipset, A7V has VIA.

• A7V has more DIMM slots.

• A7V may have more PCI slots. (I can’t remember; count them.)

• A7V (at least the particular model mentioned above) has four IDE channels on-board: two regular ATA/100 (I think) and two channels that are attached to an on-board Promise chipset which can apparently be run in either Ultra/100 or FastTrak/100 (RAID) mode. Check jumper on motherboard for details.

If you’re trying to use the latest RT, don’t use the latest HTML::Mason. I tested (I think) HTML::Mason 1.16, 1.15, and 1.14; none of them will work. Apparently 1.05 is the thing to use. If you want more information (and a possible patch with some trade-offs) search on “RT Mason 1.15″ or something.

I got lots of “hdX: lost interrupt” messages when booting the RH8 install CD in my new machine with three Highpoint Technologies HPT302 cards, otherwise known as the Rocket133 cards. Apparently the solution to this is to go get the latest Rocket133 drivers from Highpoint. Get the non-open source ones, follow the instructions in the enclosed PDF. You basically have to supply parameters like hdX=noprobe to the kernel from SYSLINUX to keep the kernel from toying with the HPT302 controllers. Then you use their drivers disk. If you want to upgrade the kernel in RH8.0 (or other versions, probably) go get the open source drivers from the page above and follow the directions for building them. Note that the open source drivers aren’t really open source, but kind of a wrapper for some library they provide only as a binary. The result of this, when using it with RH8.0, is that the driver won’t load because their library was compiled with GCC 2, but the RH kernel was compiled with GCC 3. Easiest thing to do: vi -b hpt302lib.o, %s/gcc2_compiled/gcc3_compiled/g, :wq, then build the driver. When you’ve got a working driver for your new kernel, install it in /lib/modules/KERNEL-VERSION/kernel/drivers/scsi/ and don’t forget to mkinitrd if you need it. I’ll take this time to mention that I have no idea if the GCC 2/GCC 3 thing I did above is a good idea at all — I suspect strongly it isn’t — but so far it’s working for me. Hooray for binary drivers!

Much to my dismay, you apparently can resize a RAID 5 device; i.e., expand my new RAID 5 pr0n array. You need raidreconf. While the author warns that it’s untested, and I am forced to concur (backup!), I will say that I have not read of any tales of raidreconf screwing up disks, and I have read at least a handful of success stories. Google for yourself if you don’t trust me. (I wouldn’t.)

Originally, to provide for expandability of my array in the future, I was going to seal it in LVM and call it a day. I was pleased to learn that RH8 ships with LVM. Then I found out that LVM doesn’t do RAID 5. Indeed, LVM leaves that up to the MD layer. At the time I was under the impression that there was no way to resize an MD (software RAID 5, specifically) device. So I found EVMS. It looked cool, but was very ambiguous about whether it allowed for resizing of a RAID 5 array. Then I went and found that the EVMS people, after not being included in 2.5 before the feature freeze, have kind of abandoned most of their kernel work from the sound of it. Instead they seem to be focusing on some integration work with the facilities that did make it in to 2.5 (LVM2, which seems to be more of a wrapper around something called DM, or “device manager”). So EVMS would require me to turn off the MD stuff in the kernel (per their documentation, not sure if this is strictly true though), a kernel rebuild with some stuff that may or may not be stable (probably stable), and still might leave me without being able to do what I want? Nah. Not if I can find something better. That’s when I found raidreconf. Usenet and Google saves all.

I spent most of the day today setting up my new file server. It might do more than just serve files in the future, but for now it’s just got 5x120GB drives, 3xHPT302 cards, and a willingness to serve. (Pun intended.) I’m going to seal up the software RAID 5 device in an LVM VG, so just in case things don’t work out with raidreconf for some reason in the future, I can still make another separate RAID 5 array and (theoretically) just extend the VG. Maximum VG size seems to be 2TB. Bummer? BTW, there is a utility that apparently only exists in LVM2 called pvresize (physical volume resize) which I’d need to use if I actually used raidreconf to extend a RAID device that was part of a VG. However, from posts made by the person whom I believe to be the author of pvresize, that migration from LVM1 to LVM2 is smooth, and additionally pvresize can supposedly be built for and work with LVM1. Hooray.

Another debate I’m presently having with the configuration of this box is whether to use ordered or writeback journal mode in ext3. I might try both and play with bonnie a bit to see if there’s some huge difference, but I suspect I won’t notice one in day-to-day use and will end up in ordered mode. Right now, though, the MD device decided it needs to resync, so I’m leaving it to its business. Maybe bonnie++ in the morning.

Worked on getting RT set up last night. It went pretty much as they described, except for that horrible problem with HTML::Mason (or with that version of HTML::Mason and RT, at least; I’m not at all convinced the problem is HTML::Mason’s). I’m using MySQL to back it, though it supposedly works with both PostgreSQL and Oracle as well. This is part of our evaluation of ticketing systems to be used at work, but I suspect we’re going to end up staying with RT unless it’s found unsuitable… or if something more suitable/featureful with a similarly clean code base and agreeable license (free) is found. (RT supposedly has a very clean design and is easily extendable/modifiable. I’ve seen some evidence of this in users writing add-ons and such, I think.)