February 26, 2006

UDP directed broadcast helper

I wrote a little program called pbh to forward directed UDP broadcasts along. If this doesn’t quite make sense, see my last entry. Note that the script is in Python and has only been tested on Python 2.3 (FC2) and 2.4 (CentOS 4; er, I think CentOS 4 has Python 2.4). Also note that python setup.py install probably won’t do what you want; better to python setup.py bdist_rpm and install the resulting RPM. If RPM isn’t your thing, either extract (rpm2cpio dist/pbh-0.2-1.noarch.rpm | cpio -ivdm off the top of my head) or alien it. If there’s demand I can post the “binary” RPM (pbh-0.2-1.noarch.rpm) or maybe I can even be convinced to look at making the install command work correctly. I’ve done minimal testing and it seems to work; I’ll know more when the client actually starts using it with their messaging application.

A few nice things I found while building the distribution for this program. distutils isn’t horrible. Check out setup.cfg in this distribution to see how I override the defaults for the bdist_rpm command. Note, though, that when overriding the %install script for the generated RPM, it replaces (not appends!) the default script, which means your install script needs to install everything. In my case that wasn’t a problem, but I noticed there’s no way to get the name/path of the Python interpreter from inside the provided script (bdist_rpm_install in this case).

Also, I found the fedora-rpmdevtools package which includes skeletons for RPM spec files and, more importantly to this project, an initscript (/usr/share/fedora/template.init).

February 22, 2006

Linux and directed broadcasts

A client has some instant messaging program they want to run. It’s, uh, decentralized, I guess; it requires no central server unless I’m mistaken. To discover other clients on the network it needs directed broadcast. (At least, I assume that’s the kind of activity it would undertake with broadcasts.) I consider this kind of ugly, but I’m not sure I necessarily have a better suggestion.

Now, if I were using Cisco equipment on my network, I think this would go easy. ip directed-broadcast strategically placed in routers sounds like it does the trick. However… I’m using Linux on both ends for a router. So the network looks like:

LAN A===Linux box X---Linux box Y===LAN B

X and Y are actually connected via a GRE-over-IPsec tunnel, running over the Internet of course.

Now, I’ve come to the conclusion that Linux simply doesn’t support forwarding of a packet it identifies as a broadcast. Here’s why.

First, I did some tests. Say network A uses, and network B uses I can ping from a host on A to I see the packet go over the GRE tunnel from X to Y. I can see the packet come out the GRE tunnel on Y. Y then responds with it’s 10.0.1.x address. However, Y does not forward the packet on to network B. This experiment was conducted with the help of tcpdump. /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts is 0 on both hosts. iptables should not be getting in the way (and I don’t see how you could claim it is, unless it’s blocked in the FORWARD chain and I’m pretty sure it isn’t: I don’t block anything from A to B in the FORWARD chain). I’ve tried various weird tricks to get this to work and it hasn’t yet.

This kind of makes sense if you’re familiar with Linux, and especially iptables. For B to both respond to a packet and to forward it would mean the packet would have to cross both the INPUT and the FORWARD chains; and of course we all know that doesn’t happen (uh, normally).

For further confirmation, I looked at the source for Linux net/ipv4/route.c, line 1989, you see the label brd_input:

    if (skb->protocol != htons(ETH_P_IP))
        goto e_inval;

    if (ZERONET(saddr))
        spec_dst = inet_select_addr(dev, 0, RT_SCOPE_LINK);
    else {
        err = fib_validate_source(saddr, 0, tos, 0, dev, &spec_dst,
        if (err < 0)
            goto martian_source;
        if (err)
            flags |= RTCF_DIRECTSRC;
    flags |= RTCF_BROADCAST;
    res.type = RTN_BROADCAST;

    rth = dst_alloc(&ipv4_dst_ops);

You can see that the brd_input label continues on into the local_input label after the fact that it’s a broadcast packet has apparently been noted in appropriate places. It seems plausible that some of the FIB (forwarding information base?) code determines that a packet is destined for a broadcast address, and then route.c gives it this treatment, transferring control to brd_input.

Disclaimer: I really didn’t read much of the source, and I understood only a fraction of what I did read, so don’t take this as final. I’m trying to understand more kernel source, so any comments offering clarification are very welcome.

Finally, I’ll add that Alan Cox suspects that any kind of broadcast forwarding in Linux is impossible in the context of a thread on directed broadcast on the samba-technical mailing list. Some other people confirm this. I believe I’ve seen Alexey Kuznetsov make a similar comment, though I didn’t go back and find where I read this. (Of course, in Alan’s comment he’s talking about kernel 2.2. No idea what Alexey’s comment was talking about, but I bet it was at best 2.4.)

So how do I get around this?

  • There’s the iptables ROUTE target. With this I feel somewhat confident I could force the kernel to forward the packet. (The firewall would never treat it as local input, I bet, but that’s fine since the firewall doesn’t have any of this messaging software running on it.) Problem: this isn’t available on basically any machine I have. One firewall is CentOS 4.2, which I believe lacks this target. The other is probably running a Fedora distribution, or maybe even RHL 9. Even my FC4 box only has the ROUTE target in documentation; it’s not available when you try to use it (missing userspace, if not kernel code for this target).

  • I could go nuts and change the GRE-over-IPsec to OpenVPN. Then I could bridge the networks (leaving their separate IP subnets unchanged). Using ebtables/iptables I could keep unnecessary broadcast traffic from going over the network… maybe. We’d end up with a lot of ARP going across the VPN I guess. This would be messy, and I haven’t really gotten comfortable with permanent subnet-to-subnet tunnels with OpenVPN.

  • The solution that I think I’m going to go with, as suggested by Scott Venier, is to write a helper application. From the above linked “knowledge base” sort of article, I gather I can just listen on a single UDP port and clone the packets to the local network. For an example of this approach, see dhcrelay, included in ISC dhcpd. It purpose is to relay DHCP messages (UDP broadcast) from the local network to a DHCP server on another network. I think I’ve actually used one dhcrelay process running on each end of a similar Linux IPsec-over-GRE situation to get DHCP requests from (embedded) clients on one end (like LAN A) to a server that sits on the other (like LAN B).

A helper app still isn’t exactly pretty. I guess I’ll write it in Python, or else C. Then it’s sucking up memory. I’ll probably want to package it, and I’ll have to make some startup scripts. Then I’ll have to push that package out to the other two or three sites they have.

I wonder if it’s as non-trivial as it appears to make Linux forward broadcast packets? (In a clean way, not a “hey, if we comment out this branch it’ll forward the packet, but routing will be screwed or the local machine won’t see the packet, etc.” way.) It has been pointed out to me (unconfirmed) that not providing the ability to do directed broadcast is a violation of some RFC or another.

February 16, 2006

More PmWiki setup

This is an extension of a previous post regarding PmWiki. This time I’m going to try and record in excruciating detail what I do to install PmWiki.

Bootstrapping Apache

I’m going to be setting up a Wiki on a host with a name (well, alias) of tc.codefu.org. Make sure you set up the DNS for whatever name you’re going to use first, and make sure it can be resolved from the machine that’s operating as your web server.

At the bottom of /etc/httpd/conf/httpd.conf:

### Local changes

NameVirtualHost *:80
Include conf/vhosts.d/*.conf

[darkness@linode conf]$ sudo mkdir /etc/httpd/conf/vhosts.d

Note that *:80 is necessary if you have SSL installed. Otherwise you get bitching, at least with the default RH conf. If you need to, set up the default web directory, above the Include directive in httpd.conf above. Putting it here seems to make sure it gets selected as the default VirtualHost.

<VirtualHost *:80>
    ServerName plain.old.server.name
    DocumentRoot /var/www/html

This was enough to get even /cgi-bin working. Really I’m just too lazy to look in the Apache docs to find out how it decides what virtual host to serve from in the case no Host header is passed, or in the case that the Host header is unrecognized.

Now lets configure the PmWiki host:

[darkness@linode conf]$ cat vhosts.d/tc.codefu.org.conf
<VirtualHost tc.codefu.org:80>
    ServerName tc.codefu.org
    DocumentRoot /srv/www/tc.codefu.org/root

(Note: you may also need AllowOverride FileInfo for your DocumentRoot directory if you want to use .htaccess files; see the section on “Clean URLs,” below, for the whole story.)

We’ll restart Apache later, once we actually have PmWiki installed.

Installing PmWiki

Download the latest PmWiki release. Also have the PmWiki installation instructions on hand.

mkdir -p /srv/www/tc.codefu.org/{root,dist} and wget the PmWiki distribution into that dist directory. Untar it inside root and set up a symbolic link as below.

[darkness@linode tc.codefu.org]$ pwd
[darkness@linode tc.codefu.org]$ sudo chown -R root:root root
[darkness@linode tc.codefu.org]$ sudo chmod -R u=rwX,go=rX root
[darkness@linode tc.codefu.org]$ ls -al dist root
total 244
drwxr-xr-x    2 root     root         4096 Feb 13 19:02 .
drwxr-xr-x    4 root     root         4096 Feb 13 19:02 ..
-rw-r--r--    1 root     root       233589 Nov 10 10:51 pmwiki-2.0.13.tgz

total 12
drwxr-xr-x    3 root     root         4096 Feb 13 19:02 .
drwxr-xr-x    4 root     root         4096 Feb 13 19:02 ..
lrwxrwxrwx    1 root     root           13 Feb 13 19:02 pmwiki -> pmwiki-2.0.13
drwxr-xr-x    7 root     root         4096 Nov 10 10:51 pmwiki-2.0.13
[darkness@linode tc.codefu.org]$ cd root/pmwiki
[darkness@linode pmwiki]$ sudo install -m 775 -o root -g apache -d wiki.d

At this point, if you’re on a system with SELinux (for example, Fedora Core, CentOS/RHEL 4) you’ll probably need to do something like chcon -t httpd_sys_content_t wiki.d, but that’s just off the top of my head. No SELinux on this RHL 9 box.

Now is probably as good a time as any to see if what we’ve done so far has worked:

[darkness@linode pmwiki]$ sudo service httpd reload
Reloading httpd:                                           [  OK  ]

Now I’ll try and hit http://tc.codefu.org/pmwiki/pmwiki.php. I’m glad I did this in my case, since I typoed the DocumentRoot in tc.codefu.org.conf and got a 404. Remember to check /var/log/httpd/error_log when you have problems. If everything works you should see a very normal looking PmWiki HomePage. You should probably also test editing in the SandBox to make sure pages can get written to wiki.d OK.

Continuing on, cp sample-config.php local/config.php and edit local/config.php. Most interesting variables:

  • $WikiTitle
  • $PageLogoUrl if you’re interested in a different logo
  • $DefaultPasswords['admin']: probably a good idea to set this
  • I like $EnableGUIButtons = 1 (uncomment the line) so get cute buttons while editing
  • $SpaceWikiWords = 1 if you want WikiWords to look like “Wiki Words”
  • $LinkWikiWords = 0 if you don’t want WikiWords to make a link at all
  • $WikiWordCountMax = 1 if you only want the first occurrence of a WikiWord to generate a link. Keep in mind that, on long pages (for example), this might be a big nuisance (you have to scroll up to find the first occurrence, which is the only actual link).
  • You can also disable certain WikiWords from ever making a link (maybe useful for acronyms, for example) by tweaking the $WikiWordCount associative array

To enable RSS, uncomment the section that looks something like:

# if ($action == 'rss' || $action == 'rdf') include_once('scripts/rss.php');

PmWiki >= 2.1 might or might not have this bit (it looks like rss.php has been renamed to feeds.php). See http://www.pmwiki.org/wiki/PmWiki/RSS for more information.

There are some other interesting variables dealing with “clean URLs” and file uploads. We’ll come back to them.

This is a good point at which to save config.php and test PmWiki again. If you want to test your administrator password, hit a URL such as http://tc.codefu.org/pmwiki/pmwiki.php?n=PmWiki.GroupAttributes?action=attr. This should prompt you for a password. If it accepts your password, you should be taken to a page where you can set passwords for the PmWiki group.

Clean URLs

Start with http://pmwiki.org/wiki/Cookbook/CleanUrls. It looks like they’ve recently cleaned this up, and now it’s even easier to understand.

I want something like http://tc.codefu.org/pmwiki/pmwiki.php?n=PmWiki.GroupAttributes to be http://tc.codefu.org/pmwiki/PmWiki/GroupAttributes.

[darkness@linode pmwiki]$ pwd
[darkness@linode pmwiki]$ cat .htaccess
# Use mod_rewrite to enable "Clean URLs" for a PmWiki installation.
RewriteEngine On
# Define the rewrite base.
RewriteBase /pmwiki
# Send requests without parameters to pmwiki.php.
RewriteRule ^$           pmwiki.php  [L]
# Send requests for index.php to pmwiki.php.
RewriteRule ^index.php$  pmwiki.php  [L]
# Send requests for files that exist to those files.
RewriteCond %{REQUEST_FILENAME} !-f
# Send requests for directories that exist to those directories.
RewriteCond %{REQUEST_FILENAME} !-d
# Send requests to pmwiki.php, appending the query string part.
RewriteRule (.*)         pmwiki.php?n=$1  [QSA,L]

Relevant excerpt from local/config.php:

##  $ScriptUrl is your preferred URL for accessing wiki pages
##  $PubDirUrl is the URL for the pub directory.
$ScriptUrl = 'http://tc.codefu.org/pmwiki';
# $PubDirUrl = 'http://www.mydomain.com/path/to/pub';

##  If you want to use URLs of the form .../pmwiki.php/Group/PageName
##  instead of .../pmwiki.php?p=Group.PageName, try setting
##  $EnablePathInfo below.  Note that this doesn't work in all environments,
##  it depends on your webserver and PHP configuration.  You might also
##  want to check http://www.pmwiki.org/wiki/Cookbook/CleanUrls more
##  details about this setting and other ways to create nicer-looking urls.
$EnablePathInfo = 1;

At this point I test this by going to http://tc.codefu.org/pmwiki/. Doesn’t work. From /var/log/httpd/error_log:

[Wed Feb 15 23:57:02 2006] [error] [client] Directory index forbidden by rule: /srv/www/tc.codefu.org/root/pmwiki/

Turns out I needed to modify /etc/httpd/conf/vhosts.d/tc.codefu.org.conf, which now looks like:

<VirtualHost tc.codefu.org:80>
    ServerName tc.codefu.org
    DocumentRoot /srv/www/tc.codefu.org/root

    <Directory /srv/www/tc.codefu.org/root>
        AllowOverride FileInfo

service httpd reload and I’m back in business.

One final thing: I want http://tc.codefu.org/ to go to http://tc.codefu.org/pmwiki:

[darkness@linode pmwiki]$ cat /srv/www/tc.codefu.org/root/.htaccess
RewriteEngine on
RewriteRule ^/?$ /pmwiki/ [R,L]

That should do the trick.


Now, if you also want to set up uploads, go ahead and check out my previous web log entry on configuring uploads in PmWiki. I don’t need them right now, so I’m not going to set them up here, or say much about them.

One thing I notice I failed to mention in that post is: if you want to set upload_max_filesize too high in /etc/php.ini, I think you may also need to make sure post_max_size is large enough to accommodate the larger upload size.