February 22, 2006

Linux and directed broadcasts

A client has some instant messaging program they want to run. It’s, uh, decentralized, I guess; it requires no central server unless I’m mistaken. To discover other clients on the network it needs directed broadcast. (At least, I assume that’s the kind of activity it would undertake with broadcasts.) I consider this kind of ugly, but I’m not sure I necessarily have a better suggestion.

Now, if I were using Cisco equipment on my network, I think this would go easy. ip directed-broadcast strategically placed in routers sounds like it does the trick. However… I’m using Linux on both ends for a router. So the network looks like:

LAN A===Linux box X---Linux box Y===LAN B

X and Y are actually connected via a GRE-over-IPsec tunnel, running over the Internet of course.

Now, I’ve come to the conclusion that Linux simply doesn’t support forwarding of a packet it identifies as a broadcast. Here’s why.

First, I did some tests. Say network A uses 10.0.0.0/24, and network B uses 10.0.1.0/24. I can ping from a host on A to 10.0.1.255. I see the packet go over the GRE tunnel from X to Y. I can see the packet come out the GRE tunnel on Y. Y then responds with it’s 10.0.1.x address. However, Y does not forward the packet on to network B. This experiment was conducted with the help of tcpdump. /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts is 0 on both hosts. iptables should not be getting in the way (and I don’t see how you could claim it is, unless it’s blocked in the FORWARD chain and I’m pretty sure it isn’t: I don’t block anything from A to B in the FORWARD chain). I’ve tried various weird tricks to get this to work and it hasn’t yet.

This kind of makes sense if you’re familiar with Linux, and especially iptables. For B to both respond to a packet and to forward it would mean the packet would have to cross both the INPUT and the FORWARD chains; and of course we all know that doesn’t happen (uh, normally).

For further confirmation, I looked at the source for Linux 2.6.15.4. net/ipv4/route.c, line 1989, you see the label brd_input:

brd_input:
    if (skb->protocol != htons(ETH_P_IP))
        goto e_inval;

    if (ZERONET(saddr))
        spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK);
    else {
        err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst,
                      &itag);
        if (err < 0)
            goto martian_source;
        if (err)
            flags |= RTCF_DIRECTSRC;
    }
    flags |= RTCF_BROADCAST;
    res.type = RTN_BROADCAST;
    RT_CACHE_STAT_INC(in_brd);

local_input:
    rth = dst_alloc(&ipv4_dst_ops);
                .
                .
                .

You can see that the brd_input label continues on into the local_input label after the fact that it’s a broadcast packet has apparently been noted in appropriate places. It seems plausible that some of the FIB (forwarding information base?) code determines that a packet is destined for a broadcast address, and then route.c gives it this treatment, transferring control to brd_input.

Disclaimer: I really didn’t read much of the source, and I understood only a fraction of what I did read, so don’t take this as final. I’m trying to understand more kernel source, so any comments offering clarification are very welcome.

Finally, I’ll add that Alan Cox suspects that any kind of broadcast forwarding in Linux is impossible in the context of a thread on directed broadcast on the samba-technical mailing list. Some other people confirm this. I believe I’ve seen Alexey Kuznetsov make a similar comment, though I didn’t go back and find where I read this. (Of course, in Alan’s comment he’s talking about kernel 2.2. No idea what Alexey’s comment was talking about, but I bet it was at best 2.4.)

So how do I get around this?

  • There’s the iptables ROUTE target. With this I feel somewhat confident I could force the kernel to forward the packet. (The firewall would never treat it as local input, I bet, but that’s fine since the firewall doesn’t have any of this messaging software running on it.) Problem: this isn’t available on basically any machine I have. One firewall is CentOS 4.2, which I believe lacks this target. The other is probably running a Fedora distribution, or maybe even RHL 9. Even my FC4 box only has the ROUTE target in documentation; it’s not available when you try to use it (missing userspace, if not kernel code for this target).

  • I could go nuts and change the GRE-over-IPsec to OpenVPN. Then I could bridge the networks (leaving their separate IP subnets unchanged). Using ebtables/iptables I could keep unnecessary broadcast traffic from going over the network… maybe. We’d end up with a lot of ARP going across the VPN I guess. This would be messy, and I haven’t really gotten comfortable with permanent subnet-to-subnet tunnels with OpenVPN.

  • The solution that I think I’m going to go with, as suggested by Scott Venier, is to write a helper application. From the above linked “knowledge base” sort of article, I gather I can just listen on a single UDP port and clone the packets to the local network. For an example of this approach, see dhcrelay, included in ISC dhcpd. It purpose is to relay DHCP messages (UDP broadcast) from the local network to a DHCP server on another network. I think I’ve actually used one dhcrelay process running on each end of a similar Linux IPsec-over-GRE situation to get DHCP requests from (embedded) clients on one end (like LAN A) to a server that sits on the other (like LAN B).

A helper app still isn’t exactly pretty. I guess I’ll write it in Python, or else C. Then it’s sucking up memory. I’ll probably want to package it, and I’ll have to make some startup scripts. Then I’ll have to push that package out to the other two or three sites they have.

I wonder if it’s as non-trivial as it appears to make Linux forward broadcast packets? (In a clean way, not a “hey, if we comment out this branch it’ll forward the packet, but routing will be screwed or the local machine won’t see the packet, etc.” way.) It has been pointed out to me (unconfirmed) that not providing the ability to do directed broadcast is a violation of some RFC or another.