March 18, 2003

Perl scoping

OK, so I’ve just had a long discussion about why this breaks:

[darkness@darkbook darkness]$ perl -Mstrict -w
eval 'package Foo; our $bar = 1';  die if ($@);
eval 'package Foo; print "$barn"';  die if ($@);
Variable "$bar" is not imported at (eval 2) line 1.
Global symbol "$bar" requires explicit package name at (eval 2) line 1.
	...propagated at - line 2.
Turning off strict, of course, makes everything better:
[darkness@darkbook darkness]$ perl -w
eval 'package Foo; our $bar = 1';  die if ($@);
eval 'package Foo; print "$barn"';  die if ($@);
1
The solution to make this work is to use use vars instead of our it seems:
[darkness@darkbook darkness]$ perl -w -Mstrict
eval 'package Foo; use vars qw($bar); $bar = 1';  die if ($@);
eval 'package Foo; print "$barn"';  die if ($@);
1
Why is this? It is because Perl’s scoping rules were apparently written by a commission of Schizophrenic toddlers coming down off of a two week bender, and later edited by Charles Manson and the Jackson 5. This excerpt from perldoc -f our is helpful:
       our EXPR
               An "our" declares the listed variables to be valid
               globals within the enclosing block, file, or
               "eval".  That is, it has the same scoping rules as
               a "my" declaration, but does not create a local
               variable.
That page also has two examples, one of which I found particularly interesting:
               An "our" declaration declares a global variable
               that will be visible across its entire lexical
               scope, even across package boundaries.  The pack�
               age in which the variable is entered is determined
               at the point of the declaration, not at the point
               of use.  This means the following behavior holds:

                   package Foo;
                   our $bar;           # declares $Foo::bar for rest of lexical scope
                   $bar = 20;

                   package Bar;
                   print $bar;         # prints 20
Now check out the top of perldoc vars, as in use vars (I’m on Perl 5.6.1 BTW):
       NOTE: The functionality provided by this pragma has been
       superseded by "our" declarations, available in Perl v5.6.0
       or later.  See the our entry in the perlfunc manpage.
My ass our provides the same functionality, as we’ve demonstrated above. For this reason I’ve decided on the following concepts of Perl scopes:
Lexical scope
Best given in the quote from perldoc -f our, this scope is either a “block, file, or ‘eval’.” This is why my original attempt using eval followed by another eval failed: the variable was being declared in two different lexical scopes.
Package global scope
This scope appears to only be modified by use vars. If anyone else can give me a better example, I’m listening. These variables are visible using the fully-qualified variable name, $namespace::variable. The above example which “crosses package boundaries” doesn’t work with a variable declared by use vars.
There was this third scope I had decided existed, which I called the “package local scope.” I said this because of the following:
[darkness@darkbook darkness]$ perl -Mstrict -w
package Foo;
my $bar = 1;
package Blee;
print "$Foo::barn";
Name "Foo::bar" used only once: possible typo at - line 4.
Then I remembered the bit about lexical scoping, though, so check this out:
[darkness@darkbook darkness]$ perl -Mstrict -w
package Foo;
my $bar = 1;
package Blee;
print "$barn";
1
So there goes that notion: our “package local” is visible from outside the package. It’s local to the lexical scope. OK, so I guess that means that a variable declared with our is visible to code from outside the lexical scope (though you probably need a fully-qualified name?) whereas a variable declared with my is visible only to code that operates in the same lexical scope as the my declaration. Maybe? Now what the fuck does this mean:
[darkness@darkbook tmp]$ cat test1.pl
use strict;  use warnings;

our $foo = 123;
[darkness@darkbook tmp]$ cat test2.pl
use strict; use warnings;

require "test1.pl";
print "$main::foon";
[darkness@darkbook tmp]$ perl test2.pl
Name "main::foo" used only once: possible typo at test2.pl line 4.
123
[darkness@darkbook tmp]$ perl -pi -e 's/main:://g' test2.pl; cat test2.pl
use strict; use warnings;

require "test1.pl";
print "$foon";
[darkness@darkbook tmp]$ perl test2.pl
Global symbol "$foo" requires explicit package name at test2.pl line 4.
Execution of test2.pl aborted due to compilation errors.
While this might not match my articulation thus far, it matches the behavior my brain is currently expecting. My one problem is this: if you use strict but have two files that declare variables in the same package, you can’t easily access variables from file two when they’re declared in file one. I guess it just seems fucked up. With use warnings in effect you can’t even use the fully-qualified name without getting some complaints. Moral of the story: use package statements for every file that you’re going to want to access members of later. I’ll go ahead and throw in the added bonus: you shouldn’t use $a and $b in Perl code. Ever. This is because sort uses them:
               If you're using strict, you must not declare $a
               and $b as lexicals.  They are package globals.
               That means if you're in the "main" package and
               type

                   @articles = sort {$b <=> $a} @files;

               then "$a" and "$b" are "$main::a" and "$main::b"
               (or "$::a" and "$::b"), but if you're in the
               "FooPack" package, it's the same as typing

                   @articles = sort {$FooPack::b <=> $FooPack::a} @files;
I was doing my original tests with $a and $b which resulted in weird behavior like this:
[darkness@darkbook tmp]$ perl -Mstrict -w
eval 'package Foo; our $a = 1';  die if ($@);
eval 'package Foo; print "$an"';  die if ($@);
1
[darkness@darkbook tmp]$ perl -Mstrict -w
eval 'package Foo; our %a = (1=>2)';  die if ($@);
eval 'package Foo; print "$a{1}n"';  die if ($@);
Variable "%a" is not imported at (eval 2) line 1.
Global symbol "%a" requires explicit package name at (eval 2) line 1.
	...propagated at - line 2.
Needless to say, this really fucked me up. I couldn’t figure out why $a worked but %a didn’t. Thankfully, tiResias figured that mystery out. Lots of people had suggestions about why I was doing this-or-that. Here’s something like what I’m really doing, proving why I needed to do eval:
while (reading a line from file into $line)
	{
		... do some stuff to $line ...
		eval "package $unique_package_name; $line";
	}
So I was making sure all statements from some user file get evaluated in some unique package name. This is part of my Config::PerlConfiguration class. Thanks to heller, tiResias, and Wainstead for their discussion, comments, and guidance.

Virtual mail in Postfix

[Edited 2003-03-24: added vmail_destination_recipient_limit to main.cf.]

[Edited 2003-03-19: change postfix-virtual-tools to a directory link. Just grab the latest version out of there. I'm going to be changing this package a lot in the coming days.]

Wouldn’t it be nice if I could make an entry at least once a week? I need to get back on my old schedule.

So here’s my quick run down on what I did to get a “POP toaster” kind of setup with Postfix+userdb+Courier-IMAP. Note that while this seems to work, and I have tested it a bit, and just put it in to production for one customer, it hasn’t been tested super extensively. In addition, I might be circumventing some Postfix security checks here, if not opening up some new ones on my own. Proceed with caution, use at your own risk, blah blah blah.

I’m on RH 8.0, so I install their postfix RPM. Grab maildrop and run rpmbuild -ta on the tarball. (rpmbuild -ta magically peeks inside a tarball for a .spec file and then attempts to build a package with that spec file.) This produces a maildrop package. Two patches you might want to apply to maildrop, both written by me so use at your own risk:

  • My lower case account name lookup patch to maildrop. The problem: if you have an account name bob, and someone sends e-mail to an account named Bob, (note the uppercase) maildrop rejects the e-mail. You can think of it as being case sensitive about e-mail addresses. Solution: if initial lookup in userdb fails for a given account name, we try converting that account name to all lowercase and try to look that up. I’ve been using this patch in production for a year or so. It’s not perfect by a long shot, but it works for me, and if you’ve got a similar setup to what I’m describing here it’ll probably work for you too.

  • My EX_NOUSER patch for maildrop. The problem: the author of maildrop apparently thinks the only error maildrop should ever return is EX_TEMPFAIL (see /usr/include/sysexits.h), even under errors which should probably be fatal, such as an attempt to deliver to a user that doesn’t exist. I find this kind of bizarre given that we’ve got all these nice status codes we can return, like EX_NOUSER when a user isn’t found. I think the author of maildrop thinks the job of deciding why an error occurred should be left up to the MTA, for some reason. In any case EX_TEMPFAIL will cause Postfix to defer instead of bouncing immediately, and Postfix will have no way to check if a message is attempting to deliver to a non-existent user. This patch causes maildrop to return EX_NOUSER when you attempt to deliver to a user that doesn’t exist.

In the end, here’s my spec file for maildrop. Grab this, stick the maildrop tarball in SOURCES, and rpmbuild -ba your way to happiness.

So far we’ve got Postfix and maildrop installed. Go ahead and set up Postfix for local delivery, if you’d like. I think Postfix will actually work without reconfiguration (except perhaps the root alias in /etc/postfix/aliases) but feel free to configure for local delivery if you’d like; note that we modify mydestination below. Here’s what I use for my maildrop local delivery in /etc/postfix/main.cf:

mailbox_command = /usr/bin/maildrop -d "$USER" -f "$SENDER" "$EXTENSION"

The changes to /etc/postfix/main.cf that our virtual setup is going to require are the following lines at the end:

transport_maps = hash:/etc/postfix/transport
mydestination = $myhostname, localhost.$mydomain, hash:/etc/postfix/transport
vmail_destination_recipient_limit = 1

Note that you need to comment out any mydestination value in main.cf other than the one given above. The first line says that a per-destination transport can be selected from the map /etc/postfix/transport, a file that already exists (though it has no mappings yet) in the RH8 RPM. The second line tells Postfix to receive mail for our host name, localhost, and any domains that are going to be listed in our /etc/postfix/transport. The third line instructs Postfix (really pipe(8)) that maildrop can only deliver to one recipient at a time. Without the third line, a message addressed to multiple recipients would be called like maildrop -d foo1@bar.com foo2@bar.com foo3@bar.com, which maildrop won’t accept.

Next, we have to modify /etc/postfix/master.cf. I get the feeling that normal people don’t have any good reason to modify this, but pretend you’re super-human for a bit and add the following two lines, indentation and all, at the end of master.cf:

vmail     unix  -       n       n       -       -       pipe
  flags=Fu user=vmail argv=/usr/bin/maildrop -d $recipient -f $sender $extension

You might have noticed that this line is suspiciously similar to the mailbox_command we set above. It’s basically identical, in fact. These two lines set up a new transport named vmail. It will always be run as the user vmail (hence user=vmail) and with the user vmail‘s primary group ID. We’re going to create the vmail user right after we’re done reconfiguring Postfix.

The final change to make is to add all your virtual domains to /etc/postfix/transport. Here’s an example:

myvirtualdomain.com	vmail:

Note the vmail: at the end of the line, which means “use the vmail transport for any mail addressed to the domain myvirtualdomain.com. Once you’re done don’t forget to run postmap /etc/postfix/transport to hash it for Postfix.

Now to create our vmail user. In RH8, I recommend the invocation of:

useradd -r -d /var/spool/vmail -s /bin/false -M vmail

This creates a system user (UID /var/spool/vmail, and a shell of /bin/false which should keep it from logging in successfully. Well, that and the fact that the account never gets a password set on it, and so remains forever locked. You’ll need to create /var/spool/vmail and then /var/spool/vmail/domains after this. chown both directories to vmail:vmail. Under /var/spool/vmail/domains is where all your mail is going to be spooled. Then under /var/spool/vmail/domains create a directory for each domain you want to host. Make sure these directories are also owned by vmail:vmail. You probably also want ~vmail (/var/spool/vmail) to have permission 0770, so any local users won’t go poking around where they’re not wanted. This is a good permission value for most every directory you’re going to create under ~vmail.

At this point you’re pretty much done, or close at least. Go grab some Courier-IMAP RPMs, or maybe you have to rpmbuild -ta the tarball again — I don’t remember. I’m sure if you’ve gotten this far you’ll be able to handle it. The nice thing about Courier-IMAP in this setup is that we basically have to do absolutely nothing for it to work correctly. Courier-IMAP will automatically look to userdb (/etc/userdb*) to authenticate users and such. In my setup I wanted to keep Courier-IMAP from authenticating any local users, so I took authpam out of the authmodulelist variable in /usr/lib/courier-imap/etc/authdaemonrc. (authdaemonrc was probably copied by me from authdaemonrc.dist which is in the same directory, I believe.)

Now time to start it all up and test. First service postfix restart, then service courier-imap restart. Now hope for the best.

The one thing that pushed me towards this setup as opposed to any Postfix solutions was that I’d have a single source for user information: userdb. That all said and done, at some point I might generate a map from /etc/userdb so that Postfix can reject mail to unknown users without accepting it in the first place: no bouncing!

If you’d like to add users and such, you can check out my quite immature postfix-virtual-tools package. This has had very minimal testing, and no documentation basically. (Config::PerlConfiguration was sucked brutally out of my floundering DarkWiki project, BTW.)

I’ve been getting steadily more drowsy, so I’m going to drop off now. Please let me know if you have any questions. (I’m really going to respond to you people that have written me. I promise.)