September 26, 2007

SELinux and EPEL’s mod_fcgid on CentOS 5

I rebuilt EPEL’s mod_fcgid package with the latest mod_fcgid (2.2 I believe) with no problems. Make sure to define a macro like rhel with value 5 to get the SELinux policy module built too (echo '%rhel 5' >> ~/.rpmmacros or maybe rpmbuild --define 'rhel 5' ...).

Now, I’m using FastCGI and Apache’s suEXEC to run the FastCGIs as particular (per-site) users. So for one site my configuration looks like:

<VirtualHost www.example.com:80>
        ServerName www.example.com
        DocumentRoot /srv/www/www.example.com/root

        SuexecUserGroup www_example_com www_example_com
        ScriptAlias /trac /var/www/suexec/www_example_com/trac.fcgi
</VirtualHost>

suEXEC (as distributed with CentOS 5) will only run things under /var/www, so I created /var/www/suexec. I want to run things as the example user, lets say, so I mkdir /var/www/suexec/www_example_com and chown that directory to example:example, because suEXEC requires that. Then I make a little .fcgi file that starts my FastCGI app (for Trac as you may have noticed).

So on and so forth; none of that is interesting. Now here comes SELinux: I did install the mod_fcgid-selinux package, which automatically installs an SELinux policy module for mod_fcgid (semodule --list | grep fastcgi). As far as I can tell, you must chcon -t httpd_fastcgi_script_exec_t your FastCGI scripts; without this, I believe your script won’t have access to various pipes or shared memory or something that it needs to communicate back to mod_fcgid living in Apache. Don’t label the whole directory httpd_fastcgi_script_exec_t, just the .fcgi file. See also the README.SELinux that comes in the mod_fcgid-selinux package.

At this point, especially since I had the SELinux httpd_unified boolean on (it’s on by default), I might have been in the clear. I go to run my trac.fcgi (via /trac)… and I get an internal error. /var/log/httpd/error_log spews HTML and other errors that sound generally like “I didn’t start as a FastCGI at all, here’s me as a regular CGI, whee!” The best part? No AVC messages.

Here’s where Dan Walsh’s post on disabling dontaudit rules comes in. As he says, the cool stuff is in Fedora 8 where all dontaudit rules can be disabled, even ones in policy modules (policy modules like the fastcgi module that is installed by mod_fcgid-selinux). However, in this case, and because I rebuilt fastcgi with no dontaudit rules, I can tell you the problem is in the base policy and not in a policy module; so using semodule -b /usr/share/selinux/targeted/enableaudit.pp works just fine to troubleshoot this.

So execute the above semodule command, restart Apache (for good measure?), reload /trac, and now we get:

type=AVC msg=audit(1190783371.758:8180): avc: denied { read write }
for pid=3 2127 comm="suexec" name="[180326]" dev=sockfs
ino=180326 scontext=user_u:system_ r:httpd_suexec_t:s0
tcontext=user_u:system_r:httpd_t:s0 tclass=unix_stream_socke t

type=AVC msg=audit(1190783371.758:8180): avc: denied { siginh }
for pid=32127 comm="suexec" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:http d_suexec_t:s0 tclass=process

type=AVC msg=audit(1190783371.758:8180): avc: denied { rlimitinh }
for pid=32 127 comm="suexec" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:h ttpd_suexec_t:s0 tclass=process

type=AVC msg=audit(1190783371.758:8180): avc: denied { noatsecure }
for pid=32127 comm="suexec" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r: httpd_suexec_t:s0 tclass=process

Now, the last three messages are apparently no big deal. That first one, though, is the problem. So I run that through audit2allow, a little editing, and I come out with something like:

 policy_module(local_fastcgi, 1.0.0)

 require {
     type httpd_suexec_t;
     type httpd_t;
 };

 allow httpd_suexec_t httpd_t:unix_stream_socket { rw_stream_socket_perms };

(OK, sorry, that’s actually a significant bit of editing. I just learned about the macros that I guess you’re supposed to use, like policy_module. I found rw_stream_socket_perms in the fastcgi policy source.)

I slap that in a file like local_fastcgi.te then make -f /usr/share/selinux/devel/Makefile load (careful! I think that will build and install every policy in–and possibly every policy under–the current directory). Once loaded, I restart Apache, and now my FastCGI being run by suEXEC works.

I hope you didn’t just read this overly verbose post just to find “install the above policy module and FastCGIs running with suEXEC under CentOS 5 will work with SELinux on.” Er, sorry.

I would report this as a bug… maybe I should file it against EPEL, if they’ve got some bug tracking. Then they’ll probably end up pushing it upstream to the F7 maintainer? Which I assume is where the EPEL package was derived from. My question then is: does F7 have this problem too, or is it fixed in F7 by virtue of some other part of the policy?

The rule to dontaudit httpd_suexec_t getting at httpd_t:unix_stream_socket is actually in the core Apache policy, I think, and it seems to make a reference to how Apache should “close-on-exec.” So I’m really not even sure if this policy change should go into Apache’s policy (and have a bool?) or if it should go into mod_fcgid’s policy module (and have a bool there?).

So, does someone want to advise me as to where and against what package(s) I should file a bug? Or maybe I need to wade onto the Fedora SELinux mailing list?

September 24, 2007

Making a remote Dell PE1600SC at The Planet use software RAID

We just got another server at The Planet. They had a deal on another of the same 1600SC’s we got before: dual Xeon, but this time with dual 10k 136GB SCSI disks.

They offer CentOS 5 now. (When we bought our last one, they only offered RHEL 3, which I had to remotely upgrade to CentOS 4.) When they turn the server over to you, you’ve got /, /boot, swap, and then /dev/sdb is one big partition. My task: change it to LVM on software RAID 1.

I don’t think their web site mentions it, and there was some question about whether they still offer it, but this newest server still has a Dell DRAC III card in it, which I needed to toy with grub (when I screwed up). You could probably do without remote console access, except (1) it will be hard to see errors, and (2) I don’t recall if grub has a functionality similar to lilo -R (“for the next boot only, use boot entry N”) these days.

Here are my rough steps to complete this task. This is based partly on what I did, partly what I should have done, and it’s all from memory (granted, memories that are scant hours old). Note that you probably don’t want to do this on a “production” machine (duh), as this will require reboots and potentially horking things. I did it on a fresh install that I wasn’t afraid to have reinstalled; you’ve been warned.

  1. Partition /dev/sdb. For me, that’s 128MB /dev/sda1 as /boot, and the rest in /dev/sdb2. Both types 0xfd for RAID auto-detect. I set sdb1, which will be software RAID 1 for /boot, to be an “active” (AFAIK means “can boot off of here”) partition, in case sda1 fails.
  2. Create the software RAID devices. For /boot, this like like mdadm --create -l 1 -n 2 /dev/md0 missing /dev/sdb1. missing is the word I always forget: it says “the disk that would normally go in this slot is… well, missing.” Repeat for /dev/md1 with /dev/sdb2.
  3. pvinit, vgcreate, and then lvcreate all the logical volumes you want.
  4. mkfs or equivalent on all your new devices. For the record, I found out RH’s anaconda does something like mke2fs -j -i <inode size> <device>. I left out the -i. Other options for mke2fs are apparently read from /etc/mke2fs.conf. After mke2fs anaconda seems to do tune2fs -i0 -c0 -ouser_xattr,acl -Odir_index <device>. (This is from memory, I hope it’s right.) Don’t forget to e2label everything too. RHEL 5’s mkswap apparently takes -L to set a label, which you can use in /etc/fstab.
  5. Mount up your new, empty filesystem(s) somewhere. I made and used /mnt/sdb.
  6. Edit grub.conf. Note on my system, /etc/grub.conf was not symlinked to the actual file, /boot/grub/grub.conf. If you see this I don’t see why you shouldn’t rm /etc/grub.conf && ln -s ../boot/grub/grub.conf /etc/grub.conf like I did.

    In grub.conf you want to make a copy of your currently booted stanza. In this copy, change the root (hd0,0) to be root (hd1,0), and also change the root=LABEL=/ in the kernel line to point at something like root=/dev/vgmain/lvroot (or wherever your root FS is).

    Also, for people like me on a DRAC III, you’ll want to comment out any bootsplash or hiddenmenu lines, you don’t need any serial console (I commented out both serial and terminal lines for GRUB and also removed console=... parameters to the kernel). You may want to bump up the timeout. Additionally, DRAC III users will need to add i8042.dumbkbd=1 to the kernel “command line.”

  7. I don’t think this is necessary… but I did add (hd1) /dev/sdb to /boot/grub/device.map and I also re-ran /sbin/grub-install.

  8. Copy everything from the running filesystems to the new filesystem(s). I used cd /; find . -xdev -print | cpio -pvdm /mnt/sdb/ to do this, and then the same with /boot. Jumping ahead some steps, after I finally got the new arrays booted, I don’t think this copied the SELinux labels right, which is no good. I encourage you to find a command that does copy the SELinux labels. Maybe star can help? Or cp -ax? If you use my method, go ahead and touch .autorelabel in the root of your new filesystem, which should cause SELinux relabelling at boot.
  9. chroot to your new FS. mount /proc && mount /sys.
  10. Edit /etc/fstab on the new FS to reflect your new layout. I note that a fresh CentOS 5 install using LVM on RAID doesn’t use LABEL=... syntax for selecting devices, so I’d put actual device names for everything on software RAID or LVM in column 1 of fstab. Note that I never got mkinitrd to run correctly (i.e. include the LVM stuff) as long as I was using LABEL=/ for my / partition.
  11. This is kind of weird: I think you need the device nodes hanging around to ensure that mkinitrd works right in the next step, so run start_udev. The weird part: this is going to kill the running udevd and replace it with one whose / points to your new root. I was able to finish my work and reboot my system without udevd running on my real root; hopefully your mileage won’t vary.
  12. If you’re using LVM like me, you need to run vgmknodes now, to make things like /dev/vgmain/lvroot (example name).
  13. Now it’s time to run mkinitrd. rm the existing initrd in /boot and run mkinitrd with the -v switch. Make sure it pulls in lvm (and look for lvm.conf too).
  14. Theoretically you’re now ready to go, and it’s time to reboot. (Last chance to touch /.autorelabel.) killall udevd; umount sys && umount proc && exit to get out of the chroot. umount the rest of the stuff for fun, maybe vgchange -a n and then mdadm --stop all your new RAID devices (since I’m paranoid and not sure if RHEL would do this for me, since they weren’t there at boot), and reboot.
  15. Hopefully you can now select your new GRUB entry at boot, and boot from your new disks. (If you touched .autolabel you’re in for an extra reboot.) Once it boots, log in and make sure mount looks right, with your new LVM/RAID devices. Also check /proc/swaps to make sure the right swap device is being used.
  16. Once you’re sure you’re OK, and you don’t want anything off the old disk, sfdisk --dump /dev/sdb | sed 's/sdb/sda/g' | sfdisk /dev/sda to copy the partition table from sdb to sda.
  17. mdadm --manage --add the sda partitions to the RAID devices.
  18. Wait for resync (i.e. bounce on cat /proc/mdstat).
  19. Now go back to grub.conf, take out your extra entry (the one using (hd1,0)). Change the old entry (in my grub.conf I now only had one entry) to use the real root device path, so (ex.) root=/dev/vgmain/lvroot) instead of root=LABEL=/.
  20. Make sure /etc/fstab looks right still. Again, I recommend against using LABEL= syntax for anything on software RAID or LVM.
  21. I re-ran mkinitrd with -v, more than anything just to make sure it worked right. (Again: if your / is on LVM, make sure you see things like lvm.conf being put on the initrd. I didn’t the first time I did this, and it turned out to be because I was using LABEL=... syntax in /etc/fstab for /.)
  22. Optional: I wrote a GRUB boot sector to sdb as if it were the first disk in the system. Run grub, device (hd0) /dev/sdb, root (hd0,1), setup(hd0). You can check that GRUB is now on sdb with something like dd if=/dev/sdb bs=512 count=1 | strings | grep GRUB and you should see GRUB as output.
  23. Reboot (and pray, perhaps).

Once you’ve rebooted this last time you’ll probably want to look at mount to make sure everything is still where it should be, check your swap is on, and check /proc/mdstat to make sure the arrays are OK. In my case, and slightly to my surprise, everything was OK.

Sorry to the folks at The Planet if my turning off serial console stuff in GRUB/the kernel screws them up. (Feel free to drop me a line and I’ll turn it back on and update these instructions.)

September 14, 2007

Python drive-by

Making simple tree data structures out of things like lists and dicts in Python always pains me. In my head other languages, like maybe ECMAScript or Lua, have better syntax for this. Here’s an example of what I might write to set up a particular data structure:

day = {}
day["date"] = "date object here"
day["title"] = "string data here"
sections = day["sections"] = [dict(title="section title")]
jobs = sections[0]["jobs"] = []
jobs.append(dict(title="Google",
                 url="http://www.google.com"))
jobs.append(dict(title="Yahoo!",
                 url="http://www.yahoo.com"))

Perhaps pprint can make it easier to see what I’ve done here:

{'date': 'date object here',
 'sections': [{'jobs': [{'title': 'Google', 'url': 'http://www.google.com'},
                        {'title': 'Yahoo!', 'url': 'http://www.yahoo.com'}],
               'title': 'section title'}],
 'title': 'string data here'}

You might want to argue that I made bad choices WRT that syntax, since the pretty printed version above probably looks better. Here’s an alternative, I guess:

day = {"date": "calculate date object here",
       "sections": [{"title": "section title",
                     "jobs": [{"title": "Google",
                               "url": "http://www.google.com"},
                              {"title": "Yahoo!",
                               "url": "http://www.yahoo.com"},
                              ],
                      },
                     ],
       }
# Can't assign this in-line, unless I calculate "date" into a local
# variable first.
day["title"] = "this is based on " + day["date"]

It’s not really fair to say “just write it like pprint has.” pprint has a few advantages, such as having everything that it needs to put in its rendered data structure up front, and also being a computer. As a human writing this data structure, I think of the title before I think of putting in the jobs for example; pprint alphabetizes the keys, so it puts jobs first, which means you don’t potentially have }]}]} at the end of the structure. Also note that I needed one of the values in the structure to compute another; pprint already had that value when it went to render the data structure.

Also, I may need to think about switching from double quotes to single quotes. To my overly picky mind, they now look a little “cleaner.” My use of double quotes can be traced back to when I was frequently programming in C, but is not helped by the fact that '' and "" behave differently in languages such as Perl and the Bourne Shell. (C also makes/made me do slightly weird things in Perl and sh, such as writing 'x' and "xxx" with different quote types.)

So I went crazy and made a class which is currently called DataObject. It’s actually a somewhat disgusting set of wrappers over dictionaries, but check out the syntax:

day = DataObject()
day.date = "date object here"
day.title = "string data here"
day.sections[0].title = "section title"
day.sections[0].jobs.new_child(title="Google",
                               url="http://www.google.com")
day.sections[0].jobs.new_child(title="Yahoo!",
                               url="http://www.yahoo.com")

To me this is vastly more readable (and easier to write too), and it works just like the data structure I made above with Python’s built-in types. You can also ask for a clone of the data using built-in types by calling day.to_native() (it operates recursively).

I think there may be some weird side-effects, like weird exceptions that happen when you make a typo on a “key” (since attributes are mapped to keys) and a new object springs into existence. I’m going to try using it a bit more before I pass judgment on whether or not it’s a useful idea.