May 18, 2004

Upgrading from RH9 to FC2

I popped in an FC2, hit upgrade, and let it do its thing… for a really long time. I think I have my CD-ROM and HD on the same IDE channel, and I think my CD-ROM is slow. Neither helped, I’m sure.

First thing I notice: X won’t start. I’m using an NVIDIA video card, and I had heard that 4K stacks were enabled in FC2′s kernel, thus breaking the (not open source) NVIDIA drivers. This is, to my knowledge, still not fixed — though I haven’t checked Bugzilla. So I switch the Driver "nvidia" section to read Driver "nv", killall gdm-binary (not being able to service gdm restart or killall gdm is a little annoying) and I’m back on my way.

Immediately get XKB errors. Find bug #120858. Remove XkbRules line from /etc/X11/XF86Config. Don’t just change it to XkbRules "xorg" as some comments suggest; that didn’t work for me, at least. XKB errors disappear. I guess they didn’t get the patch in FC2 before release.

My gnome-panel configuration is apparently gone. I have one icon other than the red hat (why is it still a red hat? Shouldn’t it be a… fedora… uh… of some sort?) which seems to be for e-mail, which points to evolution, which is MIA. Delete icon, accidentally click it, set gnome preference for e-mail client to mutt, fear what might happen if I ever use that preference by accident, really delete icon. Reconfigure panel. Notice monitor needs to have its settings adjusted as I just put the time off the right edge of the screen, readjust, hope I don’t have to do this a lot.

Start firing a few things up. Looking good, my LDAP/Kerberos configuration looks intact. Postfix running, SpamAssassin running. Wonder if Pyzor works? It runs. Wonder what version of Python FC2 ships? Oh look:

[darkness@morgase darkness]$ rpm -qf /usr/bin/python2.3
python2.3-2.3-1pydotorg
python-2.3.3-6

I don’t know exactly why, but I expected Anaconda to remove any packages whose dependencies couldn’t be satisfied, or warn me, or something. This did not happen. Best I have is an upgrade.log in root’s home directory with a bunch of lines like Upgrade Dependency: Needs (('apt', '0.5.5cnc6', 'fr1'), ('librpm-4.2.so', None), 0, None, 0), automatically added that seem to hint at which packages it artificially satisfied dependencies for. Lament.

Browse a bit in Mozilla. Notice mouse wheel up does not scroll up, but instead seems to simulate at least a middle click and a right click (yes, both). Lament. Stop Postfix since I don’t know for sure e-mail is working right now.

Regarding the mouse wheel problem, find bug #115147. Try suggested fix in /etc/X11/XF86Config (not the kernel parameter… yet). Close everything and restart X. Not working. Try adding suggested kernel parameter and reboot. Result as seen during boot:

psmouse: unknown protocol type 'ExplorerPS/2'

So, not the right answer. Dig in kernel sources, find the correct invocation is more like psmouse.proto=exps. Reboot. Not fixed. Andy asks what my ZAxisMapping is. It looks normal… in both cases. Somehow I have two mice configured. Now I’m not saying I blame Anaconda for this. Maybe I had that screwed up in there to begin with. So I change the second mouse configuration (I was only changing the first before) to ExplorerPS/2, restart X, and it works. Hooray.

Now, what were the other problems I was fixing, again?

Oh, right, lets get this Python issue straightened out. (Side note: things are slow. Hm, why does hdparm -i /dev/hda seem to not indicate any mdma or udma mode active? hdparm -X udma2 /dev/hda and lets see if that sticks.) rpm -e python2.3 python2.3-devel, things work. Much rejoicing.

Now what about those other packages with failed dependencies? Particularly APT, which will help me find other broken dependencies (don’t know if yum will) and apcupsd. Lets start with APT. I guess I’ll get APT by… using yum. Dig around on the Internet to reconfigure the repositories I was using in yum. Here’s my yum.conf for FC2. I actually added Livna.org while I was in there because I hear they have some nice NVIDIA driver packages. Not yet, of course, since no working NVIDIA 3D driver is available for FC2.

yum-arch -d might help you out WRT chasing down broken dependencies but I can’t quite figure it out. So rpm -e apt; yum install apt for me. Now apt-get -f install yields:

0 upgraded, 0 newly installed, 237 removed and 0 not upgraded.

<Carl’s voice>Yeah, no, we’re not gonna do that.</Carl’s voice> So instead I’ll go through upgrade.log and pick out the RPMs named in the beginning (as above).

apcupsd: I built this myself, but it looks like Dag has an RPM of it. Will it fit the bill? Backup /etc/apcupsd, rpm -e apcupsd-std-rh9, yum install apcupsd. Copy in my apcupsd.conf. Start it up, wait a minute or so, connected to the master. Yay.

Lets tackle the NVIDIA stuff. Oh joy:

[darkness@morgase darkness]$ sudo rpm -e NVIDIA_kernel NVIDIA_GLX
error: Failed dependencies:
        libGL.so.1 is needed by (installed) wine-20030408-1rh8winehq
        libGL.so.1 is needed by (installed) uqm-20030720-0+codefu
        libGL.so.1 is needed by (installed) mplayer-1.0-0.2.pre2.fr
        libGL.so.1 is needed by (installed) xawtv-3.81-6
        libGL.so.1 is needed by (installed) libquicktime-0.9.2-fr0.pre1.2
        libGL.so.1 is needed by (installed) xorg-x11-libs-6.7.0-2
        libGL.so.1 is needed by (installed) xorg-x11-Mesa-libGLU-6.7.0-2
        libGL.so.1 is needed by (installed) chromium-0.9.12-25
        libGL.so.1 is needed by (installed) tuxracer-0.61-26
        libGL.so.1 is needed by (installed) xorg-x11-tools-6.7.0-2
        libGL.so.1 is needed by (installed) kdegraphics-3.2.2-1
        libGL.so.1 is needed by (installed) xscreensaver-4.14-5
        libGL.so.1 is needed by (installed) kdeartwork-3.2.2-1
        libGL.so.1 is needed by (installed) freeglut-2.2.0-11
[darkness@morgase darkness]$ rpm -q --whatprovides libGL.so.1
NVIDIA_GLX-1.0-4363

Shouldn’t something else provide that? yum search libGL.so.1 gives no love, but rpmfind.net does. rpm -e --nodeps NVIDIA_GLX so we can install xorg-x11-Mesa-libGL which supposedly has the file we want. Uh, hold on:

[darkness@morgase darkness]$ sudo rpm -e NVIDIA_GLX --nodeps
  --- WARNING!!
      xxx.libGLcore.a.rpmsave
      xxx.libglx.a.rpmsave
 --- The above files are apparently saved files from a
 --- non-NVIDIA distribution (possibly MESA) that were
 --- put aside while the NVIDIA rpm was installed.
 --- They are now being restored.

That can not be good. Sure enough:

[darkness@morgase darkness]$ locate libGLcore.a
/usr/X11R6/lib/modules/extensions/libGLcore.a
[darkness@morgase darkness]$ rpm -qf /usr/X11R6/lib/modules/extensions/libGLcore.a
xorg-x11-6.7.0-2
[darkness@morgase darkness]$ rpm -V xorg-x11
/etc/security/selinux/file_contexts: No such file or directory
..?.....    /usr/X11R6/bin/Xorg
S.5....T    /usr/X11R6/lib/modules/extensions/libGLcore.a
S.5....T    /usr/X11R6/lib/modules/extensions/libglx.a

That’s Probably Bad™. BTW, those were my NVIDIA RPMs. There were never really super clean. Why doesn’t yum have an option to re-install an RPM? Sigh. (Ooh, find yum provides which I should have been using instead of yum search.)

In the midst of doing this I click in the workspace switcher. Gnome Bug #122068 rears its ugly head. Perhaps this was put in too late to be in Fedora? Or Gnome 2.6? I have no idea. I like that four month delay, though. I find it odd more people weren’t clamoring for this to be fixed. I compiled my own libwnck packages that hopefully have this fixed. You will have to restart X for the change to take effect.

Reinstalled xorg-x11 and those libraries are back to normal. My NVIDIA_kernel package must be uninstalled with --noscripts, I guess because it demands /sbin/update-modules, which has disappeared. I am not worried.

yum wants to pull spamassassin-tools from Dag. No, we’re not doing that either. (I have my own build of SpamAssassin.) exclude=spamassassin*. Otherwise I think yum is going to be able to update all my other dependency problems, chiefly with Multimedia stuff like mplayer. I’m going to set it running then go shower; I’m going to be late for dinner.

Oh, one more thing: my monitor keeps kind of flipping off for a second then coming back. It happens irregularly, maybe once every half hour or something? Weirdness. NVIDIA, please hurry up with those drivers.

Oh, wait, e-mail. Forgot about that. Had to rebuild TMDA but that was easy (rpmbuild -ta tmda-1.0.3.tgz and install resulting RPMs). Other than that everything looks OK.

Update: I forgot about editing menus. But don’t go trying to copy /etc/gnome-vfs-2.0/default-modules.conf.with-menu-editing to default-modules.conf like you’re used to. You’ll get no menus (and supposedly you still can’t edit anything). Apparently menu editing is disabled in FC2.

Also, to get rid of whining like /etc/security/selinux/file_contexts: No such file or directory install the policy package.

May 17, 2004

The state of the VPN

I was tasked with connecting Windows XP to a company’s Linux firewall. I thought this would be easy to do, and it probably isn’t too hard – especially now that I’ve played with all sorts of VPN stuff so much. It is easy to do — unless you want to do it “right” by my (unrealistic?) standards.

PPTP is still out it seems. I gather that PPTP v2 is still the latest. The security of PPTP’s communications is reportedly tied to the quality of the password used. It also sounds like there are issues with multiple PPTP clients behind the same NAT. I read that Windows Server 2003 or some such cruft “solves” this problem somehow. I am not sure what the situation is in Linux regarding NAT.

There are a plethora of other projects doing VPN, like OpenVPN, CIPE, vtun, etc. I don’t much care for these. I fear their security; didn’t Bruce Schneier and/or Counterpane do a round-up of these lesser known VPNs and claim to find security concerns galore? Moreover I don’t think many of them — if any — have support for XP. (Off the top of my head, I know CIPE doesn’t.)

Then there’s IPsec. Bruce didn’t speak real highly of IPsec either IIRC, but he did conclude that it was the current best practice. I think I’ve covered here the disappointment involved when you try to use Windows 2000 or XP’s built-in IPsec. Chief problem: doesn’t work with a dynamic IP. To use it in tunnel mode you must enter the IP of your local machine at some point, as well as the IP of the secure gateway on the other end. The best workaround for this is probably using the ebootis VPN tool. It takes a FreeS/WAN-looking configuration in and sets your IPsec policy in Windows based on that configuration. It can take the liberty of determining your dynamic IP and including it in the policy for you. Downside: not exactly super integrated, which might not be ideal for people on the road who aren’t computer savvy. I suspect that if you don’t explicitly deactivate the IPsec policy this tool creates it will remain in effect — possibly past a reboot (I did not test this, mere speculation) — thus screwing up access when the user comes back to the office. I also hate popping up console windows for users in Windows. They’re just not used to them, and I suspect it makes our solution look less professional… for whatever that’s worth.

There’s the built-in L2TP over IPsec in Windows. That looked promising enough, though difficult to set up on the server side. This time I used strongSwan and did apply the NAT-T patch. I’ll discuss what happens before you install the NAT-T patch first, though.

The symptom of running without NAT-T when you are in fact behind NAT is seeing the L2TP daemon keep getting EAGAIN from recvfrom() (strace your L2TP daemon). I’m pretty sure this is a result of an invalid UDP checksum. The UDP checksum is invalid because it is based on the source and destination IP, computed by the client who is behind NAT, and then encrypted as part of IPsec transport mode — which is the mode Windows uses automatically. Then the NAT box gets a hold of the packet, changes the IP header, but can’t see inside the packet to correct the UDP checksum. You could disable UDP checksums to get around this problem… but doesn’t that sound like a bad idea? You don’t have this problem with NAT and IPsec in tunnel mode, unless I’m mistaken, because in tunnel mode the encrypted payload is an entire IP packet rather than just the payload of the surrounding (unencrypted) IP packet. In other words, in tunnel mode, the secure gateway discards the IP header information that the encrypted payload arrived with, instead decrypting the payload and getting an entire IP packet out of that. You can disable the automatic IPsec policy used by L2TP over IPsec and then create your own policy, but you can’t create a tunnel mode policy for the reasons stated above; it still won’t work with a dynamic IP.

So, you install the NAT-T patch. Don’t forget to install it in Windows as well! I thought my regularly updated Windows clients already had the NAT-T patch installed, but my XP SP1 box did not and I had to install it from Windows Update Catalog (or something like that; I don’t claim to be a Windows expert). Now you get all sorts of new log messages talking about how NAT was detected and NAT-T is being used. Don’t forget to open up UDP port 4500 on your secure gateway as well; NAT-T also apparently uses IPSEC over UDP on port 4500. So you authenticate just fine to strongSwan, but then… surprise!

May 13 17:35:17 hostname pluto[2354]: "road-warrior"[2] 1.2.3.4:4500
#35: NAT-Traversal: Transport mode disabled due to security concerns

Whee! For information on why this is disabled, see http://www.vpnc.org/ietf-ipsec/02.ipsec/msg01260.html and http://lists.openswan.org/pipermail/users/2004-April/000498.html. The second link in particular mentions that the solution is to have FreeS/WAN (or whatever derivative thereof) assign an IP to the client, ignoring the problem of client support for doing that which I seem to recall being rejected (believe I saw this on a Fedora list where they talk about bringing Openswan into the distribution). You can probably enable transport mode with NAT-T by defining I_KNOW_TRANSPORT_MODE_HAS_SECURITY_CONCERN_BUT_I_WANT_IT. That define quite rightly doesn’t exactly make you feel comfortable. The real risk, as I understand it (see the first link at the beginning of this paragraph) is similar to the problem I described with PPTP previously: only one used behind a given NAT can have a VPN connection with the secure gateway. In our case the secure gateway is the router and firewall for the network, so it means that if someone behind a given NAT establishes a VPN connection, any data from that NATed source address will be sent through the tunnel. So if two guys from the same company are at a small office where everything is NATed to the same address, I do believe only one of them will be able to contact the home office network — period, not just as a VPN connection. (Example: one can’t make a VPN connection while the other uses web mail.) I don’t see this as a particularly rare case, but maybe I’m off my rocker. The author of the NAT-T patch is more concerned with the fact that packets which aren’t destined for the machine which has set up the tunnel might be sent to it anyway, causing an unintentional compromise of information. I consider this less risky in my situation, since I’m only establishing VPN connections with authenticated clients anyway; if you can establish a connection into the VPN you can wreak plenty of havoc without having to resort to getting behind the same NAT as another person and sniff their communications into the office.

So unless you’re willing to restrict your clients to not being behind NAT, or unless you’re willing to accept whatever security consequences (and horrible troubleshooting madness) come as a result of enabling an IPsec mode which has specifically been disabled by people who probably know more than you about IPsec, L2TP over IPsec is out. L2TP by itself is possible but not recommended. I suspect its security is no better and probably worse than PPTP. I don’t know of anyone using bare L2TP.

So now you’re left with doing straight IPsec with a third party client. This means you have to pay for software. I have previously used and enjoyed SSH Sentinel. However this product has been bought and discontinued by SafeNet. So instead you go get SoftRemote. SoftRemote really used to suck, despite the fact that Cisco’s VPN client was based on it. I’m happy to say that it sucks far, far less these days. One quirk: it bundles ZoneAlarm, and requires it in XP. This is bizarre: in 2000 you install SoftRemote, which also installs ZoneAlarm. You shut down ZoneAlarm. You can make a VPN connection with SoftRemote no problem. In XP you install SoftRemote, and along comes ZoneAlarm. You shut down ZoneAlarm. You try to make a VPN connection with SoftRemote, and it pops up an error telling you that you have to have ZoneAlarm running. ZoneAlarm looks OK, and it’s probably a good idea in the end anyway: if someone compromised a client’s PC and started that VPN connection, they’re right past your firewall. SoftRemove seems to work, and it does NAT-T. It has a nice little system tray icon where you can connect or disconnect VPN connections. One more thing to mention: the NAT-T patch for FreeS/WAN mentions the use of vhost specifications for rightsubnet to keep users from hijacking a valid LAN address, so be sure to pay attention to these instructions.

So my clients will be doing straight IPsec ESP tunnel mode for a VPN for now. Eventually I’d like to use L2TP over IPsec, I guess, despite the angst I have for it not working and the question of why L2TP is needed when IPsec could do the job just fine. My clients would probably appreciate it and it would make me feel better about the flexibility of Linux to meet any situation that you throw at it. Maybe I’ll work on contributing patches to one project or another to attempt to implement the suggestion Mathieu Lafon made in the post I linked above — if it’s even possible from the perspective of the built-in IPsec clients on Windows.

Oh, a couple more notes, actually. I don’t think the ipsec-tools that comes with Fedora Core 2 supports NAT-T in transport mode either. I’m not sure KAME/Racoon/whatever on *BSD supports it either. They have tunnel mode working in NAT-T and they think this is enough, to the best of my knowledge. Also, check out Jacco de Leeuw’s guides based around using FreeS/WAN with Windows L2TP/IPsec. Good stuff there and apparently current. In fact, I just noticed that document has a patch to enable NAT-T in transport mode as well as some links to where to get the NAT-T patch for Windows.

Also, I’m quite interested in anyone telling me how other devices/operating systems solve the problem of NAT-T in transport mode that I described above, if they address it at all. I know next to nothing, but I’m kind of guessing they special case L2TP over IPsec transport mode and do some kind of stateful packet filtering. Just a speculation.