February 22, 2006

Linux and directed broadcasts

A client has some instant messaging program they want to run. It’s, uh, decentralized, I guess; it requires no central server unless I’m mistaken. To discover other clients on the network it needs directed broadcast. (At least, I assume that’s the kind of activity it would undertake with broadcasts.) I consider this kind of ugly, but I’m not sure I necessarily have a better suggestion.

Now, if I were using Cisco equipment on my network, I think this would go easy. ip directed-broadcast strategically placed in routers sounds like it does the trick. However… I’m using Linux on both ends for a router. So the network looks like:

LAN A===Linux box X---Linux box Y===LAN B

X and Y are actually connected via a GRE-over-IPsec tunnel, running over the Internet of course.

Now, I’ve come to the conclusion that Linux simply doesn’t support forwarding of a packet it identifies as a broadcast. Here’s why.

First, I did some tests. Say network A uses 10.0.0.0/24, and network B uses 10.0.1.0/24. I can ping from a host on A to 10.0.1.255. I see the packet go over the GRE tunnel from X to Y. I can see the packet come out the GRE tunnel on Y. Y then responds with it’s 10.0.1.x address. However, Y does not forward the packet on to network B. This experiment was conducted with the help of tcpdump. /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts is 0 on both hosts. iptables should not be getting in the way (and I don’t see how you could claim it is, unless it’s blocked in the FORWARD chain and I’m pretty sure it isn’t: I don’t block anything from A to B in the FORWARD chain). I’ve tried various weird tricks to get this to work and it hasn’t yet.

This kind of makes sense if you’re familiar with Linux, and especially iptables. For B to both respond to a packet and to forward it would mean the packet would have to cross both the INPUT and the FORWARD chains; and of course we all know that doesn’t happen (uh, normally).

For further confirmation, I looked at the source for Linux 2.6.15.4. net/ipv4/route.c, line 1989, you see the label brd_input:

brd_input:
    if (skb->protocol != htons(ETH_P_IP))
        goto e_inval;

    if (ZERONET(saddr))
        spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK);
    else {
        err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst,
                      &itag);
        if (err < 0)
            goto martian_source;
        if (err)
            flags |= RTCF_DIRECTSRC;
    }
    flags |= RTCF_BROADCAST;
    res.type = RTN_BROADCAST;
    RT_CACHE_STAT_INC(in_brd);

local_input:
    rth = dst_alloc(&ipv4_dst_ops);
                .
                .
                .

You can see that the brd_input label continues on into the local_input label after the fact that it’s a broadcast packet has apparently been noted in appropriate places. It seems plausible that some of the FIB (forwarding information base?) code determines that a packet is destined for a broadcast address, and then route.c gives it this treatment, transferring control to brd_input.

Disclaimer: I really didn’t read much of the source, and I understood only a fraction of what I did read, so don’t take this as final. I’m trying to understand more kernel source, so any comments offering clarification are very welcome.

Finally, I’ll add that Alan Cox suspects that any kind of broadcast forwarding in Linux is impossible in the context of a thread on directed broadcast on the samba-technical mailing list. Some other people confirm this. I believe I’ve seen Alexey Kuznetsov make a similar comment, though I didn’t go back and find where I read this. (Of course, in Alan’s comment he’s talking about kernel 2.2. No idea what Alexey’s comment was talking about, but I bet it was at best 2.4.)

So how do I get around this?

  • There’s the iptables ROUTE target. With this I feel somewhat confident I could force the kernel to forward the packet. (The firewall would never treat it as local input, I bet, but that’s fine since the firewall doesn’t have any of this messaging software running on it.) Problem: this isn’t available on basically any machine I have. One firewall is CentOS 4.2, which I believe lacks this target. The other is probably running a Fedora distribution, or maybe even RHL 9. Even my FC4 box only has the ROUTE target in documentation; it’s not available when you try to use it (missing userspace, if not kernel code for this target).

  • I could go nuts and change the GRE-over-IPsec to OpenVPN. Then I could bridge the networks (leaving their separate IP subnets unchanged). Using ebtables/iptables I could keep unnecessary broadcast traffic from going over the network… maybe. We’d end up with a lot of ARP going across the VPN I guess. This would be messy, and I haven’t really gotten comfortable with permanent subnet-to-subnet tunnels with OpenVPN.

  • The solution that I think I’m going to go with, as suggested by Scott Venier, is to write a helper application. From the above linked “knowledge base” sort of article, I gather I can just listen on a single UDP port and clone the packets to the local network. For an example of this approach, see dhcrelay, included in ISC dhcpd. It purpose is to relay DHCP messages (UDP broadcast) from the local network to a DHCP server on another network. I think I’ve actually used one dhcrelay process running on each end of a similar Linux IPsec-over-GRE situation to get DHCP requests from (embedded) clients on one end (like LAN A) to a server that sits on the other (like LAN B).

A helper app still isn’t exactly pretty. I guess I’ll write it in Python, or else C. Then it’s sucking up memory. I’ll probably want to package it, and I’ll have to make some startup scripts. Then I’ll have to push that package out to the other two or three sites they have.

I wonder if it’s as non-trivial as it appears to make Linux forward broadcast packets? (In a clean way, not a “hey, if we comment out this branch it’ll forward the packet, but routing will be screwed or the local machine won’t see the packet, etc.” way.) It has been pointed out to me (unconfirmed) that not providing the ability to do directed broadcast is a violation of some RFC or another.

Comments (4)

  1. September 29, 2009
    Robert Woodcock said...

    I find myself in an uncannily similar situation. If you wound up writing such a helper app, could you please post a link to it or email it to rcw@blarg.net?

  2. October 11, 2009
    darkness said...

    I actually did write a little daemon in Python to receive and forward directed broadcasts. Unfortunately I no longer work at that job, and the source code is the property of my former employer.

    You can write something quick and dirty to do this in an afternoon. I think the hardest part was figuring out how to specify which interface to listen/send on (some Linux specific ioctls, perhaps).

  3. September 22, 2011
    Robert Woodcock said...

    I finally got around to doing this. I don’t think it was worth it, since after testing the script I found that each Softros client sends out an ACK to every client that it receives a broadcast message from (it replies with the *exact same data* it was sent). So if you have 200 clients, that’s 200^2 UDP packets flying across your WAN connection every minute.

    But in case someone else needs it, here it is:

    #!/usr/bin/perl -w
    # Get a list of all interfaces and find the local IP and netmask
    # Listen for Softros Messenger traffic (UDP port 19771)
    # If traffic is received from the local network, do nothing
    # If traffic is received from any other interface, resend it to the local
    # broadcast address and spoof the original sender IP just as if Linux had
    # forwarded the packet
    my $localnet = “eth1″;
    my $port = 19771;
    my ($localnetaddr, $localnetmask, $localbroadcast);

    use IO::Socket;
    use IO::Interface qw(:flags);
    use IO::Select;
    # NOTE NOTE NOTE NOTE NOTE This script requires a modification to
    # Net::RawIP – the stock version does not set SO_BROADCAST so sending
    # packets to a broadcast address results in -EACCESS.
    #
    # Add this to util.c’s rawsock() just below the IP_HDRINCL code:
    # if (setsockopt(fd, SOL_SOCKET, SO_BROADCAST, &val, sizeof(val)) new( Proto=>’udp’,
    LocalAddr=>’0.0.0.0′,
    LocalPort=>$port,
    ReuseAddr=>1,
    Broadcast=>1
    ) || die “Can’t bind: $@\n”;

    my @interfaces = $socket->if_list;
    for my $if (@interfaces) {
    if ($if eq $localnet) {
    $localnetaddr = ip2long($socket->if_addr($if));
    $localnetmask = ip2long($socket->if_netmask($if));
    $localbroadcast = $socket->if_broadcast($if);
    }
    }

    while ($socket->recv($buf, 10000)) {
    my ($peerport,$peeraddr) = sockaddr_in($socket->peername);
    my $peerip = inet_ntoa($peeraddr);
    # Determine if it came in on the local interface or not
    if ((ip2long($peerip) & $localnetmask) == ($localnetaddr & $localnetmask)) {
    #print “$peerip (local) said $buf\n”;
    } else {
    # Rebroadcast locally
    #print “$peerip:$peerport (remote) said $buf\n”;
    #print “resending from $peerip:$peerport to $localbroadcast:$port\n”;
    $sendsocket = Net::RawIP->new({
    ip => {
    saddr => $peerip,
    daddr => $localbroadcast,
    },
    udp => {
    source => $peerport,
    dest => $port,
    data => $buf,
    },
    }) || die “Can’t create raw socket: $@\n”;
    $sendsocket->send;
    $socket->recv($buf, 10000); # Immediately eat any data we just sent out
    }
    }

    # From http://weblogs.litmusgreen.com/ed/archives/002250.html
    sub ip2long {
    return unpack(“l*”, pack(“l*”, unpack(“N*”, inet_aton(shift))));
    }

    • September 22, 2011
      Robert Woodcock said...

      Oh – the “use IO::Select;” is vestigial and can be removed.