darkness

I’m more than a bit punchy right now from my nearly three whole hours of sleep last “night.” Please excuse me.

For something like the past 24 hours or so I’ve been working to get our customers’ new VPN up. They’ve got a bunch of sites with T1’s to the Internet. Hanging off each T1 is a Linux box with a Sangoma card and a few network cards. The Linux box acts as the router, firewall, proxy, secure gateway, and coffee maker.

I don’t really like using FreeS/WAN tunnels because for a normal setup you have to (A) define a whole bunch of them, and (B) define every IP (or IP block) that’s going to be passing through a given connection. I seem to have found a better way to do it, though, largely with the help of LW.

The way I’ve currently got it set up, two secure gateways make a host ESP tunnel with FreeS/WAN. Over that IPSec tunnel we lay another GRE tunnel. GRE tunnels are far less picky about what travels over them. Here’s a typical connection for one of the secure gateways:

conn corporate
        auto=start
        rightupdown=/usr/local/lib/ipsec/_updown-tunnel
        left=1.2.3.4
        leftnexthop=%defaultroute
        right=5.6.7.8
        rightnexthop=%defaultroute
        leftrsasigkey=corporate key...
        rightrsasigkey=branch key...

This is a simple-enough connection between the corporate (main) office and a branch office. _updown-tunnel is a little script I wrote that handles bringing up/down the GRE tunnels when the IPSec tunnels come up/down. It uses a configuration file, /etc/ipsec-tunnels.conf which has contents like this:

corporate tunnel corporate mode gre remote 1.2.3.4
corporate addr 5.6.7.8/32
corporate route 10/8
corporate route 1.2.3.4/32 table 16
corporate rule from 10.1.2.0/24 to 1.2.3.4 lookup 16

This is the equivalent of the following iproute2 commands:

ip tunnel add corporate mode gre remote 1.2.3.4 dev ipsec0
ip link set corporate up
ip addr add 5.6.7.8/32 dev corporate
ip route add 10/8 dev corporate
ip route add 1.2.3.4/32 table 16 dev corporate
ip rule add from 10.1.2.0/24 to 1.2.3.4 lookup 16

Keep in mind that 1.2.3.4 is corporate’s routeable (external) IP address and 5.6.7.8 is the branch’s routeable address. All private IPs on the VPN are in the 10/8 reserved address space. This particular branch uses 10.1.2.0/24 for its LAN side.

The first rule creates the tunnel with the endpoint being the corporate external IP. The tunnel is “bound” (that’s how I think of it) to ipsec0: when ipsec0 goes down, I suspect the corporate tunnel does as well. Of course, with FreeS/WAN I don’t really think ipsec0 “goes down” unless FreeS/WAN is stopped and the ipsec module is removed. Here’s the output of ip addr ls dev corporate from the branch in case you’re interested:

173: corporate@ipsec0:  mtu 16236 qdisc noqueue 
    link/gre 0.0.0.0 peer 1.2.3.4
    inet 5.6.7.8/32 scope global corporate

You’ll notice the @ipsec0 portion of the name. If ipsec0 disappears (service stop ipsec for example) that @ipsec0 turns into @nnn where nnn is the number that was listed in front of ipsec0 in the ip addr ls dev ipsec0 output. (corporate’s “ID number” as I’ve come to call it is 173 in the above example.) It seems, though, that the tunnel still hangs around, but with the M-DOWN flag. I suspect M-DOWN is “media down.” Even if a new ipsec0 comes back, the old tunnel will hang around and remain M-DOWN. I suspect this is because the reference to its interface is kept as this ID number and not the interface name.

Note that the tunnel device has the same address as our external interface, as does ipsec0. FreeS/WAN bitches about this, but I’ve seen no adverse impact for all its whining.

The first route in the above ipsec-tunnels.conf example says to route anything for the 10/8 network through the corporate tunnel. If the branch wants to hop to another office it can do so through corporate. Indeed, there are actually two hub offices like corporate that have a link between them, and then each hub office has the branches hanging off of it. In this manner, traffic could conceivably go from branch to hub to hub to branch. (OK, if that confused you, just move on.)

The last two are a trickier part I needed to allow me to do ping 10.1.2.254 (branch’s internal IP address) from, for example, the corporate firewall. When sending packets from the Linux boxes, they will choose the source address for the packet as the outbound interface address. In our case, this would be our external (routeable) address. This is bad, however; for example, pinging from corporate to the branch’s internal address would send a packet from 1.2.3.4 to 10.1.2.254. When the branch tries to return the packet, it will send from 10.1.2.254 to 1.2.3.4. This reply is the problem: the routing table chooses ipsec0 for the packet to be output, but FreeS/WAN drops the packet because of the seemingly bogus source address. To get around this, we set up a new routing table which has just the routes to our IPSec endpoints, and then a rule which shunts traffic from a private IP to a routeable address through this new routing table.

So far (the past 12 hours or so) this has been working just fine. The tunnels even come up/down correctly. I’ll leave the iptables rules as an exercise to the reader. Don’t forget about not filtering loopback, allowing traffic to be input on ipsec0 for IP protocol GRE, and washing behind your ears.

I set up a quick zone I called “internal” in corporate’s named and put a forwarders { 1.2.3.4; }; statement in each branch’s /etc/named.conf. This appears to work. However, I’m not sure dig and host are looking at the search internal statement in /etc/resolv.conf; or, at least, looking at them correctly:

[root@branch root]# host corporate.sc
Host corporate.sc not found: 3(NXDOMAIN)
[root@branch root]# dig corporate.sc

; <<>> DiG 9.2.1 <<>> corporate.sc
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4237
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;corporate.sc.			IN	A

;; AUTHORITY SECTION:
sc.			596	IN	SOA	NS1.SEYCHELLES.NET. dnsadmin.registryadvantage.com. 4927 21600 3600 2592000 600

;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Nov 28 01:33:03 2002
;; MSG SIZE  rcvd: 114

[root@branch root]# ping corporate.sc
PING corporate.sc.internal (10.1.0.254) from 5.6.7.8 : 56(84) bytes of data.
64 bytes from 10.1.0.254: icmp_seq=1 ttl=64 time=8.50 ms

--- corporate.sc.internal ping statistics ---
1 packets transmitted, 1 received, 0% loss, time 0ms
rtt min/avg/max/mdev = 8.508/8.508/8.508/0.000 ms

Suggestions?

In other news, I spent some quality time with pmud yesterday adding ways to bring my wireless card back up, as well as to restore fnset which somehow seems to get hosed during suspend/resume. Here’s my pmud-0.10-1b.2.src.rpm. Feel free to check out /etc/power/pwrctl.d/wireless and /etc/power/pwrctl.d/fnset.

My Powerbook locked up again yesterday. I’m not fond of that. I think I’ll be saving my work a lot more often. I suspect that it’s Linux not hardware, without any good reason. Yesterday it was an oops in the swapper task, I think. This may rule out that my USB serial port was causing problems. OTOH, it doesn’t bode well for me using Linux on this laptop in the future. Apparently it is possible to run OS X on this beast. Maybe I’ll try that some day.

Off to continue working on the remote site cut-overs. There’s two guys down there basically switching everyone to DHCP. Everything is going fine thus far, but then we’re not really doing anything over the VPN yet (corporate hasn’t been switched over). Wish me luck.