darkness

SpamAssassin has been getting some false negatives, more than when I started using it. I typically get between one and five spams in my inbox per day, instead of getting them in my spam folder where SpamAssassin-detected stuff goes. So after hearing some people on an unnamed IRC network talk about Bogofilter successes, I decided to give it a whirl.

Bogofilter itself comes in an RPM, but it requires a library called Judy which does not. Judy’s build process seems really complex, and I question the need for that. I’m in no position to question it, having hardly inspected it, but I still question it. So there. Anyway, here is my RPM .spec file for Judy. You should be able to grab the Judy sources and rpmbuild -ba (remember rpmbuild -ba not rpm -ba under RH 8.0) this to your heart’s content. I also note that the spec file was done up quickly and doesn’t do things it probably should, like patching their install scripts rather than half-assed writing my own. There is a path containing ia32 in the spec file, too, so it will only build under x86 without some very minor changes. Though I question whether the library is capable of being built under different architectures.

Anyway, with that out of the way, I decided to hook Bogofilter into my .mailfilter file. Since it needs to be trained, and I’m not sure I entirely trust it, I just set it up to add a header to the message. I also set it up such that spams identified by SpamAssassin are reported to Bogofilter as spam. Here’s the relevant portions of my .mailfilter:

# This is my Bogofilter test code.  Adds a header based on the result
# of Bogofilter.  Note that this header is strictly informational right
# now; it is not used for any filtering.
trash=`bogofilter`
if ($RETURNCODE == 0)
        {
                BOGOFILTER="yes"
        }
else
        {
                BOGOFILTER="no"
        }
xfilter "formail -I "X-Bogofilter: $BOGOFILTER""

# Filter the message through SpamAssassin.  If it's spam, hand it to
# bogofilter as spam (we can always take it back with "bogofilter -H")
# and then put it in the spam folder (a Maildir folder).
xfilter "spamc"
if (/^X-Spam-Status: Yes/:hD)
        {
                trash=`bogofilter -s`
                to mail/spam
        }

I also added some macros to my .muttrc, based on the ones recommended with Bogofilter (manually wrapped to fit on the page):

macro index cd "<enter-command>unset wait_keyn<pipe-entry>
bogofilter -sn<enter-command>set wait_keyn<delete-message>"
macro index cr "<enter-command>unset wait_keyn<pipe-entry>
bogofilter -sn<pipe-entry>spamassassin -rn<enter-command>
set wait_keyn<delete-message>"
macro index cf "<enter-command>unset wait_keyn<pipe-entry>
bogofilter -hn<enter-command>set wait_keyn"
macro index ef "<enter-command>unset wait_keyn<pipe-entry>
spamassassin -d | bogofilter -Hn<enter-command>set wait_keyn"

These commands are all from the index, as you may have noticed. A brief summary, with keys in Emacs notation:

• C-d: report the message to Bogofilter as spam and delete the message.

• C-r: report the message as spam to both Bogofilter and SpamAssassin (my install should report to Razor), and then delete the message.

• C-f: report the message to Bogofilter as ham (non-spam). This is used when training Bogofilter. Use it on any message you receive in your inbox that is non-spam.

• M-f: change the disposition of a message from spam to ham in Bogofilter. This is used when a valid message is mistakenly marked as spam by SpamAssassin, put in the spam folder, and (via my .mailfilter) automatically reported to Bogofilter as spam. This calls spamassassin -d to remove any “SpamAssassin markup” from the message before feeding it to Bogofilter. (Actually, this key is Escape-f; in Emacs that’d be equivalent to M-f though, AFAIK.)

In the process of writing the above I realized that I have a problem with my .mailfilter. The message is first filtered through spamassassin. If the message is spam, the message is then sent to Bogofilter. However, in the process of executing the xfilter command on the message, the SpamAssassin markup is now in the message. Not good. I’ll try changing the rule to read trash=`spamassassin -d | bogofilter -s` and hopefully it’ll interpret the | properly. (Need to test this.)

So far back as I can remember, Bogofilter has correctly been identifying the spam that gets erroneously dropped into my inbox as spam. It hasn’t been identifying as spam every message that SpamAssassin does, though. It seemed to be having special trouble with Base64-encoded spam — of which there is surprisingly a lot. Apparently spammers, the wily bunch they are, have been encoding text messages in Base64 or quoted printables entirely. This is to avoid simple text filters. SpamAssassin decodes these for analysis (and gives them something like 3.9 points against them for doing it), but I’m not certain whether Bogofilter does or not. I think it does, based on some web searches, but its results don’t necessarily reflect this. In any case, my results are kind of fubar because of the flaw I found above. I’ll probably have to kill the entire database I’ve built and rebuild from scratch, or at least the spam messages.

BTW, there is information on using SpamAssassin and Bogofilter together. Haven’t tried it, but it looks pretty simple. I may very well end up using this after I’ve got Bogofilter trained better.

[This will be one of three or four entries I hope to complete tonight. I haven't posted in a few days, but I have been doing stuff -- if not necessarily the most productive stuff.]

So I built a tin can waveguide antenna (or “the Cantenna” as it shall be called hereafter). I’ve got pictures of it up at http://www.codefu.org/people/darkness/cantenna/ in various stages of design and at various angles. The pictures aren’t the best; they were done with my Intel web cam and Windows 2000 Imaging. Also, if you notice, I’ve blurred out certain unimportant details (like my face) from the pictures. Pay no attention to the man behind the can. If there are problems with the pictures that you think were a result of conversion from TIF to JPEG, please let me know; I did it real quick with ACDSee, which has a decent interface for converting a bunch of files. BTW, ACDSee will also apparently handle multi-page TIF files and splitting them up in to separate files. If only I had known this, I would have used the faster method of snapping a bunch of pictures in the Intel web cam tools, then saving it to a multi-page TIF with Windows Imaging, then just split it out to separate files with ACDSee. Sigh.

Where to start? This was over the past weekend, so I might screw up some details or leave some steps out. For starters I removed the label from the can, opened it with *gasp* a can opener, emptied the contents into the Memory Hole, and then washed it with Dawn and water. This is not precisely rocket science, of course. I measured the displacement of the element from the back of the can using the calculator given on the Cantenna how-to page. For the record again, the can’s diameter was 8.5725cm, giving us a 1/4 waveguide length of 5.6412cm. This is actually calculated by me using a set of RPN scripts I wrote; the Cantenna how-to’s calculator disagrees with this value by something like 0.0508cm I believe. (BTW, RPN, despite having what I would categorize as a weird scripting language, is quite nice. I now long for an HP 48GX. I also want dc for the Palm Vx, or an HP 48 emulator. Someone please let me know if either of these exist.)

I borrowed darkho’s drill for this activity. It was a Black & Decker something-or-another with a set of DeWalt drill bits that worked swimmingly. I used a 3/32″ bit to drill the hole for the element originally, but I had to work it bigger and may have eventually resorted to the 7/64″ bit I used for the #4 (7/64″) screws. (Er, I think I used the 7/64″ bit. Maybe I used the 3/32″ bit for all of them. I started with the 5/64″ bit, which was way to small. To duplicate my experience exactly you could start with the 5/64″ and work your way up.) The can was relatively easy to drill: it didn’t bend or collapse like I suspected, though I had to put what I felt was a lot of pressure on it to actually get the bit through. (I most of this on the kitchen table, BTW.)

After drilling the hole for the little brass extrusion on the N connector, I inserted it and proceeded to try and mark the holes on the can. My fat sharpie wouldn’t fit through the holes to touch the actual can, so I tried a Pilot P-500 roller ball pen — which actually marked on the can, to my surprise. I then removed the N connector and remarked the pen spots with the Sharpie. Drilling the holes for the screws was similar to drilling the holes for the element, except the holes for the screws fell on the can’s grooves in such a way that it was difficult to keep the bit on the spot to be drilled. I suppose I should have taken Nightwolf up on his offer of a drill press here. No matter; as a result of the position of the holes with relation to the grooves on the can the connector ended up being mounted sort of twisted. The element remained in the right place it seems, but the connector had to be twisted a little crooked. You can easily see this in the pictures. After I drilled the first hole and managed a screw in it, I attached the connector with the one screw and started drilling through it, both to make sure the holes were in the right place and to try and steady the bit in the grooves again.

So the connector is crooked, the screws are at weird angles, but it all seems to be OK WRT the actual element inside the can, which I presume is the important part. Once I read the suggestion of putting the heads of the screws inside the can and the nuts on the outside, I made the holes as bit bigger so I could easily slip the screws in from the inside with my hands. Now it was time to solder some wire into the extrusion from the N connector. I happened upon a 50′ spool of Red 12 gauge copper wire at Lowe’s for something like $3; quite a bargain. I took off a five inch or so piece, stripped it down, and stuffed it into the extrusion on the connector. It was a pretty good fit: not tight, but not real loose. I measured the height up to 1.21″, or 3.07cm, and tried to cut it off carefully at that height. (Incidentally, while I know nothing about the calculations used in the design of a waveguide antenna, I suspect this is 1/4″ wavelength of the 2.4GHz band used by 802.11b.) I actually took a rubber-covered hammer to the piece of wire at some point to try and get it as straight as possible. I then heated up ye ole’ soldering iron and put the wire in place. I was pleased with the straightness of the end result, after some very small adjustments by bending the wire a bit. Make sure to wait for the solder to dry.

Almost done now. The hole I made for the element wasn’t quite enough to accommodate the bit of solder I had globbed on to it, so I had to use the drill again to widen the hole a bit. Holding the N connector in place with the element inside the can I inserted the screws from the inside of the can and attached the bolts. Voila, Cantenna complete. Now for testing.

At my disposal I had an older Orinoco Silver card and a brand new Cisco Aironet 350 LMC card. I have pigtails for both (from FA&B of course). I was running the tests on my Powerbook, running Yellow Dog Linux 2.3, their kernel 2.4.19-4a, orinoco_cs 0.13a, and either the version of airo_cs that came with that kernel or the latest one from CVS (I can’t remember which). I also used wavemon to monitor the card. Oddly my Orinoco card/drivers weren’t reporting signal strength. So I ended up using the Cisco 350 card for the remainder of the tests. Note that I did have both antennas on (it has two MMCX connectors) enabled, and in retrospect maybe that’s a bad thing. I always figured that “two antennas are better than one”, but I know in some cases pointing the actual antenna connector at the AP did seem to increase the signal strength noticeably. I was testing to an Orinoco RG-1000 AP — which seems kind of shitty to me, at least WRT features and interface, and might be my biggest problem in the first place. The AP was on channel 11.

First I just tested in my room about 3′ from the AP to see if I could notice a difference in wavemon when I pointed the antenna at the AP. I could. I remember the jump being significant, probably between +10dBm and +15dBm or something. (BTW, decibels still confuse me, so sorry if my terminology is all off here. It might be just “a gain of +15dB”, but the signal is reported to me in dBm, so I’ll keep it that way.) Then I went and sat the AP on top of the couch in the living room, about 2.5′ off the ground I’d estimate, with the lights (i.e., the front) facing the wall in anticipation of my coming tests. Testing in the living room (probably like 8′-10′ from the AP) I observed basically the same results. Then I took the antenna outside. I could still get signal, but it was faint. I also knew there was an AP somewhere else in the complex. I was able to pick up the signal, and I think I pinpointed it to the building across from the back of our building. I had some trouble pinpointing this signal from our third story apartment, though. It would seem to lose signal at random. I eventually think I found one position where the antenna was sitting on a railing that kept the beacon packets coming.

BTW, the Cisco card seems to “lose” the signal from time to time. I can detect this time because of a lack of change in the “link quality” metric in wavemon (and from the wireless extensions in general, I presume). This figure is usually jumping all around. When my signal level was low and I stop seeing a lot of changes in the numbers for a second or two, I presume this to mean a loss of signal. BTW, the link quality seems to be kind of not what I expected with the Cisco card/drivers: it seems that the lower the number, the better the link.

I then tried to talk to the AP from ardent’s apartment, which is the final goal of all this. I estimate his apartment to be no more than 250′ from mine in a straight line (he’s first floor, I’m third). No luck. No signal at all, in fact. I thought for sure this antenna would be able to power through the three or four apartments or whatever to get to the AP which was sitting on the wall closest his apartment, but no. I couldn’t even get it from the hall outside his apartment. I had to go out to the free space behind his apartment to get even a meager -89dBm signal.

I was quite disappointed. We tried a few different orientations, but no luck with them. I decided to test the antenna a bit like the guys tested their original designs. We went out to the parking lot in front of our apartment where we had a clear shot all the way to the other side of the apartment complex, by the apartment office. I ran an extension cord from the alternator in my car to the middle of the street where we hooked up the AP. Then I took my GPS to measure distance, my laptop, and the Cantenna. I walked to the end of the complex and the GPS told me it was about 550′. (It was reporting accuracy to 10′ or 20′ or something like that.) When I first pointed the antenna to the AP, I couldn’t get any signal at all to it! Then I radioed ardent (we were using my walkie-talkies again; me == dork) and asked him to pick the AP up off the ground. Boom, I had a signal. It was hover around -85dBm IIRC — though it may have been -75dBm. I wish I had taken better notes. Of the notes I did scribble down long after the experiment I wrote -85dBm first, so that’s the number I’ll stick with.

Needless to say, at half the distance from the AP the guys in the home-brew wireless antenna shoot-out were using, I was getting less signal than they were. I even had a better can, IIRC (their diameter was supposed to be too big for an optimal design). I tried to follow all instructions and was very careful about it, but the antenna wasn’t good enough. I think I need to e-mail the Cantenna guy(s) for some advice on what I did wrong.

Feeling a bit puzzled I decided to whip out my FA&B 5dBi omni-directional mobile antenna and see what it could do. Quite very surprisingly, it allowed us to get a signal to my AP — which was unmoved — from inside ardent’s apartment! We found that holding it in some odd places and odd angles helped. For example, we got a good result holding it near the bottom of his porch door and tilted back about 45 degrees. The tilting back makes sense if you look at the radiation pattern of the antenna, I think. Also we found that blinds made a great big difference. I suspect window screens might also.

So in the end we might just end up with some omni-directional antennas. Or maybe ardent will get an omni-directional antenna and I’ll get one of FA&B’s cheap (but +15dBi or something like that) parabolic antennas, which I believe are pretty damn directional. I could point the antenna at his omni and hopefully get good signal. We’ll have to look in to this more, and think about the investment (probably ~$60).

This concludes my adventure in Cantenna land. If you have questions on what I did, or comments to make it work better, please e-mail me.

Had to stop by work on the way home from the S.O.’s this morning: a customer web server crashed last night, and it’s not in the rack so I couldn’t MasterSwitch it. Incidentally, this is the machine I recently installed Oracle on. It’s Red Hat 7.2 if memory serves. It complained by hitting max files open, I think, while running updatedb. I get the feeling it never recovered after that. It wouldn’t allow logins because I don’t think it could spare an FD to open the PAM libraries. I’m not quite sure why something didn’t exit (like updatedb) and free up some files. Makes me wonder if increasing the max number of open files on the system (I think I can do that) will help, or if that would just mean it’d take longer for the runaway process to eat up all FD’s on the system.

Then I was off to Harris Teeter, a local grocery store chain. I needed a can for my Cantenna. Snag: didn’t have a ruler. Instead I took the bottom of a piece of paper, which had a known size of 8.5″. Then I halved it, and halved it again, so I could get 4.25″ and 2.125″. I also took the bottom of a 5″ piece of paper and used that to mark 0.75″ off the end of the 4.25″ fold of the first sheet of paper. That should have given me a nice way to measure 3.5″. I also used the Cantenna how-to to calculate the 3/4 Guide Wavelength for diameters of between 3″ and 4″ in 0.25″ increments and wrote down those lengths.

When I got to the store, I couldn’t find any suitable cans. I checked soups, vegetables, weird canned items. Finally I found some canned ravioli from Chef Boy-ar-dee. I wouldn’t eat the shit to save my life (plus it had “beef” in it) but it seemed to be 4″ in diameter and was the tallest can I had found. If memory serves, the can was 5.5″ high, and the 3/4 Guide Wavelength for a diameter of 4″ is 5.15″. Feeling good I checked out with the Chef and headed home.

Once I got home… I still didn’t have a ruler. I found that darkho’s key chain had a mini-Leatherman on it, and that had a ruler along the back of it. It was awkward, but I got about the same measurements. (There was actually a lot more “oh, wait, no, I think I was way off measuring” and then “oh, no, I was right the first time” in between, but I’m trying to skip that.) Then I plugged 4″ in to the Cantenna how-to just to revisit the numbers, and I realized the TM01 cut-off frequency was too low at 4″ in diameter. Doh! I forgot to look at that number when I was transcribing the lengths onto the piece of paper I was using. Now I had to voyage out again.

In between here I’ll slip in that I got my car worked on. Something like $218. They said my brakes were fine (I thought they weren’t – must just be my driving) but I needed a new “Serpentine belt” and a coolant flush. Whatever. Let me pay my blood money and keep my free car running, please.

Anyway, now to Lowe’s, because I need screws for my N connectors. Oh, did I forget to mention that FA&B got my connectors out to me in record time? The box said three day, but I think it was at most two days, and quite possible one day. From FL to NC, sure, but still impressive. I like those guys, I think. Anyway, the Cantenna how-to says he used “#6×1/4″ stainless” nuts and bolts. After walking around Lowe’s screw aisle for a long time I found some #6 machine screws at 1/2″ in a pack of 50 or 100 or something obscene. I looked at them, I looked at the hole in the connector in my pocket… it didn’t quite seem right. So doing my best secret agent impression, I slipped a screw out of the box and tried to fit it in the connector. (My whole time at Lowe’s was spent half looking out of my eye anyway, since I kept pulling this tiny metallic thing out of my pocket (the connector) and putting it back again and again. I kept waiting for someone to accuse me of stealing it, then me having to explain how they likely didn’t even sell them there, what the connector was for, where I got it, what I was doing, etc. No such excitement happened, of course.) The #6 screw was a bit too big, or the threads were wrong. No-go there, so I tried the next size down, #4. I’ll take this time to mention that I had to guess that #4 was smaller in diameter than #6, and prove this experimentally. (Here’s a screw chart I think.) Lowe’s screws were kind of disorganized, so it took some time. I think I’ve determined that a “#6×1/4″” screw is #4 size, 1/4″ long. (I don’t think they had anything 1/4″ long.) I ended up with some #4×1/2″ that looked like they might fit, and the ones I got came with nuts in the package. Haven’t tried them in the connector yet, though; I didn’t have the courage to try and get away with opening a bag of screws. Too risky!

Then back to Harris Teeter with my new tape measure (from Lowe’s) to try and find a can again. I realized I missed the coffee aisle. So now I’m running through Harris Teeter with a tape measure that I’m sticking on every can I can find. Somehow I didn’t feel like this was going to get me picked up by store personnel, even though I bet it was more suspicious than some mystified guy spending 30 minutes in the screw aisle taking some strange metal object out of his pocket. Anyway, all coffee cans I saw were no-go: all their diameters were too large (greater than 3.66″, which is about the max according to the JS on the Cantenna how-to page; BTW, 2.88″ should be about the minimum diameter — and that’s a fucking tall can). So I scour every aisle, re-measure a bunch of cans, and still no success. I can say this: cans are usually either 2.5″ (I think; measure a soup can and remind me), 3.375″, or 4″. Nothing in the store that I looked at fits (including beer cans), so I’m ready to head out and settle for the Chef, when in desperation I check out the baby stuff aisle. Eureka! Baby formula. (Please note that I paid something closer to $3.50 before tax, if memory serves.) This can was the usual 3.375″ in diameter if memory serves, and I’m pretty sure it was greater than the needed height of 6.66″ (or 6.71″ according to the Cantenna JS page). However… now I seem to recall thinking it was 6.5″ high, which would be bad. I’ll check it again tomorrow. Anyway, it’s probably a “more perfect” can than the Chef, maybe. I can always try both. Once you get the Chef, you can’t give him back. (Well, I guess you can, but I don’t want to return the stuff. Maybe I’ll let someone else give it to a canned food drive if they want.)

Triumphantly I bought one can of baby formula and headed out. Tomorrow (today, technically) I build the Cantenna hopefully. If anyone wants some baby formula you’d best be at my apartment by 1130 tomorrow to collect it, else it goes down the sink. Going to borrow drills from some people. I want a laser pointer to use in aiming it for testing, too. I hope to get some cooperation from someone tomorrow to help me test it in open space across the apartment complex or something. I’ll probably try and use my GPS to measure the distance between the two points (by walking). Not terribly accurate perhaps, but it’ll work more or less.

I had to do some sick stuff with our name servers tonight to make sure that zones get updated both in the internal and externals views. Basically the notifies still get sent to the routable (Google gives more hits for “routable” than “routeable”; my spelling is hereby corrected) addresses, so I had to have the Linux firewall DNAT them to proper internal addresses. It was either this or add also-notify and notify explicit statements for every zone. Also, I have one zone that is dynamically updated, so to get it to synchronize between internal and external zones I had to actually have it send notify messages to itself (on a different IP address that the other view catches). Weirdness. I think this part, at least, would all be much more sane with djbdns.

Time for bed. I am so not a vampire lately.

Busy day today, actually. First I was all slothful and slept in. Not good. I did receive a bunch of calls while I was sleeping though, so from like 0930 on it wasn’t exactly continuous sleep.

Today I had a guy installing a T1 tell me I was to use PPPoE encoding on it. I quote from an e-mail (the answers are grafted on to the end of my questions; I’ve inserted some whitespace there to make them more readable and hidden IPs):

- How many channels should we be using on the T1?  10
- At which channel does the allocation begin?  (This is probably
  channel 1.)  15 -24
- What is our encapsulation type?  pppoe
- If Frame Relay, what is our DLCI?  DLCI 16 => VC 121 VP 1
- Is WAN side numbered or unnumbered?  yes
- If WAN side is numbered, what is our WAN side IP? x.x.x.110
255.255.255.252
- What is our default gateway's IP?  x.x.x.109
- What are our LAN side IPs, if any?  x.x.x.112-115 same subnet

Not the easiest thing to decipher. Finally someone relayed to me, through the customer, that the encapsulation was to be “IETF”. I took this to mean Frame Relay, and set it up accordingly. Guess what? It no worky tonight. On top of that, I managed to kill (at least) the network on the Linux router this was being set up on. Hopefully just a simple reboot tomorrow morning will repair it.

Turned up the VPN in Charlotte (as opposed to Columbia a couple weeks ago) tonight. All but one site is up. The lingering site isn’t on-line because we forgot to bring over a patch cable long enough to hook up the Ethernet earlier in the week when we dropped the equipment over, and never rectified it. So I’m getting a call on that early tomorrow too.

Going to have a busy day tomorrow. Time for bed.

Got woken up by a client early this morning and that screwed my whole sleep schedule up.

Moved my two name servers over to new IPs. All that’s left to do now is the firewall itself, and oddly I don’t think that’ll be a very big problem. All done remotely, too. Maybe I’ll end up posting my checklist for changing IPs on a Red Hat box some day (or at request).

Did various other things today. Nothing much to talk about, really. Someone gave me a good idea: put my SSH identity files on my USB disk with my PGP keys.

Talked with ardent about doing wireless between our two apartments. I suspect I may end up building the cantenna. Need to get the N connectors from FA&B, but everything else sounds pretty stock. Sounds easy to fuck this antenna up, but we’ll see. I’m concerned about figuring how to aim it, too. I’m considering that our apartments should be parallel to each other, and then using a laser pointer to determine the angle between my apartment and his. Should allow for more accurate aiming. Luckily we have the same floor plans in our apartments, and our two coat closets face each other on what is possibly the least obstructed path between our apartments. He wanted to find out more about Linksys (or other brands) AP bridging, and if an AP doing bridging can serve normal wireless clients too. I was dubious. Need to do research.

Hopefully more of interest tomorrow.

[Amusingly, posting this entry failed originally because I forgot to change the IPs for the name virtual hosts in Apache's configurations. perl -pi -e 's/foo/bar/g' is your friend.]