Rebuilding Ubuntu Dapper’s netboot image

2007 February 10
by darkness

I’m trying to install Ubuntu 6.06 (Dapper Drake) Server onto a Dell PowerEdge 1950 server. (Note: I suspect much of this will help users of Dell PowerEdge 2950 servers as well.) The problems:

  1. I have a network installation environment set up at home, but the bnx2 driver for the server’s Broadcom NetXtreme II (something like that) NICs isn’t on the netboot disk.
  2. Apparently there may be some problems with the LSI MegaRAID SAS controller that’s in there. I’m not using hardware RAID, so I’m not sure if I was ever going to hit these problems in the first place. (I never got to my first boot, since I needed the network card drivers to complete my network install.)
  3. This wouldn’t be a problem but for the aforementioned problems: no usb-storage module means I can’t slap some kernel modules on my USB flash disk and load them from the installer’s shell on tty2.

So I want to rebuild a netboot image with bnx2, usb-storage, and maybe the new drivers will fix up megaraid_sas too. I don’t actually need usb-storage if I’ve got the correct modules already available on the netboot image, but it’s easy to add it in and I’ll use this netboot image in the future. I’ve figured out how to do this, partially by trial and error. (Thankfully I’ve written this after the fact, so as to leave out the “error” part of “trial and error.”)

Note that I’m installing the i386 version of Dapper, not x86_64. Also note that, if you want to follow along with what I’ve done here, you’ll need an Ubuntu Dapper installation. I originally tried this on an Edgy installation but debian-installer wouldn’t build the netboot image because of a version conflict with Edgy’s libnewt (Edgy had libnewt0.52, Dapper wanted libnewt0.51, and I couldn’t get them both on the system.) I’ve made a Dapper chroot on an Edgy installation I have running under VMware on my FC5 box.

Basic plan of attack:

  1. Rebuild net-retriever package to look at the dapper-proposed distribution in addition to the normal Dapper repositories.
  2. Use debian-installer (d-i henceforth) to build a new netboot image. This produces a netboot.tar.gz that’s just like the one distributed in the Dapper release, except with an initrd that includes updated kernel modules and a modified net-retriever.

Work already done for us

netboot, and probably all debian-installer (d-i henceforth) images, uses some kind of slimmed down version of the Debian packaging system called “udeb.” I believe these packages are (barely) managed by the udpkg program.

There is a dapper-proposed distribution that has kernel udebs for 2.6.15-50, which includes the bnx2 module. (2.6.15-26 is what is included on the 6.06.1 release’s netboot image.) Note there is also a usb-storage-modules udeb which we can include on our netboot image to get the usb-storage module. We’re going to make our netboot image use these updated kernel udebs from dapper-proposed, which makes our job a lot easier (no kernel rebuilding/repackaging).

Changes to net-retriever and making a repository

A few words on what doesn’t work: As I’ll cover later, d-i builds the netboot image for us, and you can tell it what distribution(s) to pull things like kernel udebs from. So at first I just told d-i to pull udebs from dapper-proposed in addition to dapper (and dapper-updates and dapper-security). However, after I booted from my new netboot image, the installer bitched that it couldn’t find more kernel modules to download. This happened because, although the kernel on the netboot image was pulled from dapper-proposed, the installer was still set to download more kernel modules only from the dapper distribution which only included kernel modules for 2.6.15-26. Then I found the preseed option mirror/udeb/suite where you can set the “suite” (distribution) from which udebs are pulled. It defaulted to dapper, so I changed it to dapper-proposed and then the installer could find all the 2.6.15-50 kernel modules. The problem with this is that the installer then only looks at dapper-proposed: it couldn’t find the next bit of the installer to download and run, since that was only in the dapper (or maybe dapper-updates or dapper-security) distribution. To say it another way: dapper-proposed only had the new kernel; the rest of the installer components haven’t aren’t in dapper-proposed, presumably because no one has felt the need to repackage proposed updates to the rest of the installer components.

So we need the installer to look at both dapper and dapper-proposed. It’s unfortunate that d-i doesn’t seem to give us a way to do this. However, Ubuntu has made some modifications to d-i to make it look at <dist>-updates and <dist>-security in addition to <dist> (where <dist> in this case is dapper). So if we can add <dist>-proposed to that list, we’ll be in business.

For the case of netboot, this checking of extra repositories by Ubuntu is performed in net-retriever. As you may have guessed, installs done from CD-ROM or other sources might use different “retrievers”, so if you’re not using netboot to do a network install, this probably won’t apply to you.

In addition to patching and rebuilding net-retriever, we also need to put it in a repository, since later on d-i’s netboot build procedure will use APT to pull in all the udebs it needs. (OK, maybe we don’t need to put it in a repository; there are probably easier ways to do it. This way seemed to be the “rightest” way to do it, though.) So the steps for our work with net-retriever are:

  1. Fetch net-retriever sources.
  2. Patch net-retriever.
  3. Rebuild the net-retriever packages.
  4. Put the new packages in a repository.

Rebuilding the packages

(Side note: I wanted to set up a proxy for APT to use (Squid caching packages). I stuck Acquire::http::Proxy "http://proxy:3128"; in /etc/apt/apt.conf.d/90local-http-proxy, but I noticed it wasn’t hitting Squid. After you make a change to your APT configuration, it seems using apt-config dump to look and see if the setting you want is there is a fine idea. In my case, I found out files in /etc/apt/apt.conf.d with .conf extensions are apparently not liked. After renaming the file, apt-config dump | grep Proxy showed my setting, and APT was using the proxy server.)

As referenced in a previous entry about rebuild an Ubuntu package I have a set of packages I like to install before I start doing package rebuilding sort of tasks, shown below. I’ll be doing as much work as possible with a non-root user (darkness).

$ sudo apt-get install dpkg-dev gcc libc6-dev dpatch fakeroot \
                       devscripts ccache debhelper

Next lets get the net-retriever sources.

$ mkdir dpkg-build
$ cd dpkg-build/
$ apt-get source net-retriever
Reading package lists... Done
Building dependency tree... Done
Need to get 32.7kB of source archives.
Get:1 http://us.archive.ubuntu.com dapper/main net-retriever 1.09ubuntu2 (dsc) [703B]
Get:2 http://us.archive.ubuntu.com dapper/main net-retriever 1.09ubuntu2 (tar) [31.9kB]
Fetched 32.7kB in 0s (915kB/s)
dpkg-source: extracting net-retriever in net-retriever-1.09ubuntu2
dpkg-source: unpacking net-retriever_1.09ubuntu2.tar.gz
$ cd net-retriever-1.09ubuntu2/

Next we need to patch net-retriever:

--- net-retriever.orig  2007-02-07 22:11:50.000000000 -0500
+++ net-retriever       2007-02-07 22:11:59.000000000 -0500
@@ -110,7 +110,7 @@

        # TODO: Ubuntu-specific hack to fetch -updates and
        # -security; is there any way to make this more generic?
-               for codename_extra in "$codename" "$codename-updates" "$codename-security"; do
+               for codename_extra in "$codename" "$codename-updates" "$codename-security" "$codename-proposed"; do
            if [ "$codename_extra" = "$codename-security" ]; then
                thishost="$sechostname"
                thisdir=/ubuntu

Update the change log with dch -i, since we’ll want our package’s version to be higher than the one available in Dapper. Fill it in as appropriate:

net-retriever (1.09ubuntu2.1) dapper; urgency=low

  * Also provide packages from the -proposed variant distribution.

 -- darkness <darkness@caliginous.net>  Wed,  7 Feb 2007 22:12:26 -0500

[...]

When you’re done you’ll see something like:

"debian/changelog.dch" 712L, 23825C written
dch warning: your current directory has been renamed to:
../net-retriever-1.09ubuntu2.1

This is no problem.

I forgot: you need a GPG key

Now I’ll take a short detour. Down the line, when you try to use your new net-retriever udeb, the d-i process that builds the netboot image will, by default, have a fit if your package/repository isn’t signed. There are a few ways around this:

  • Sign the package and repository.
  • Edit the d-i scripts to prevent APT from checking package signatures (I think this is something like APT::Authentication "false".) This might be a bad idea if you think it’s a possibility that one or more of the udebs you will download may be tampered with.
  • You could even turn off authentication in /etc/apt.conf or similar, but beware the above caveat and remember to turn it back on later! Don’t let automated upgrade processes run while you’ve got this set either, unless you really trust your repositories.
  • As I mentioned before, there are probably easier ways to get your net-retriever udeb into netboot than creating a repository (I think there’s something like localudebs), and I suspect they won’t check to see if your package is signed.

I’ve never taken the easy way out. Never! So I’ll choose that first option, the hardest one. This will require me to make a key and make it known to APT. In fact, I’ll add the key–temporarily–into the system keyring, because I don’t feel like modifying the d-i scripts to set APT::GPGV::TrustedKeyring to a temporary keyring. Note that the key you generate will only be used on your build system; it will not be distributed nor included in our final product, the netboot image. (If you have a key already you can feel free to use it, with the caveat that you’ll need to get your password into mini-dinstall’s release signing script somehow. By default, you do this by writing your password into a file on disk–which is bad for any key you care about.)

Also note I’m secreting away the GPG keys we’ll use into a separate directory, so as not to interfere with my user’s ~/.gnupg and files therein. (Again, if you want to use an existing key, I suppose you might want to skip the part where I change GNUPGHOME.)

$ export GNUPGHOME=~/dpkg-build/.gnupg
$ gpg --gen-key
gpg (GnuPG) 1.4.2.2; Copyright (C) 2005 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

gpg: directory `/home/darkness/dpkg-build/.gnupg' created
gpg: new configuration file `/home/darkness/dpkg-build/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/darkness/dpkg-build/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/darkness/dpkg-build/.gnupg/secring.gpg' created
gpg: keyring `/home/darkness/dpkg-build/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 1024
Requested keysize is 1024 bits
Please specify how long the key should be valid.
     0 = key does not expire
  <n>  = key expires in n days
  <n>w = key expires in n weeks
  <n>m = key expires in n months
  <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: darkness
Email address: darkness@caliginous.net
Comment:
You selected this USER-ID:
"darkness <darkness@caliginous.net>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++.++++++++++++++++++++++++++++++.+++++++++++++++++++++++++.++++++++++++
++++++++++++++++++++++++++++.+++++++++++++++..+++++++++++++++>..++++++++++.
...>+++++..............................................+++++


Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 281 more bytes)
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.++++++++++.++++++++++..+++++++++++++++++++++++++.+++++++++++++++++++++++++
+++++++++++++++++++++++++.++++++++++.+++++++++++++++++++++++++>+++++.....>+
++++.....................................................<+++++............
.......>+++++............<+++++..........>+++++............................
............<+++++.................................................>+++++<+
++++...+++++^^^^^

gpg: /home/darkness/dpkg-build/.gnupg/trustdb.gpg: trustdb created
gpg: key 7D9B4A32 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   1024D/7D9B4A32 2007-02-08
  Key fingerprint = C6D6 7D3F ABB2 8299 93F2  8349 1D6C 58F7 7D9B 4A32
uid                  darkness <darkness@caliginous.net>
sub   1024g/A90719AF 2007-02-08
$ gpg -a --export darkness@caliginous.net | sudo apt-key add -
OK

Note that debsign (by way of debuild, which we run below to build the net-retriever package) uses the name/e-mail you gave in your debian/changelog entry to find your secret key; you can specify a different key ID with -k I believe.

Now that we’ve got a GPG key squared away, lets build our modified net-retriever package:

$ debuild
dpkg-buildpackage: source package is net-retriever
dpkg-buildpackage: source version is 1.09ubuntu2.1
dpkg-buildpackage: source changed by darkness <darkness@caliginous.net>
dpkg-buildpackage: host architecture i386
 fakeroot debian/rules clean
dh_testdir
dh_testroot
dh_clean
 dpkg-source -b net-retriever-1.09ubuntu2.1
dpkg-source: warning: unknown information field `Xb-Installer-Menu-Item' in inpu t data in package's section of control info file
dpkg-source: building net-retriever in net-retriever_1.09ubuntu2.1.tar.gz
dpkg-source: building net-retriever in net-retriever_1.09ubuntu2.1.dsc
 debian/rules build
[...]
dpkg-deb: building package `net-retriever' in `../net-retriever_1.09ubuntu2.1_al l.udeb'.
warning, `debian/download-installer/DEBIAN/control' contains user-defined field `Installer-Menu-Item'
dpkg-deb: ignoring 1 warnings about the control file(s)
dpkg-deb: building package `download-installer' in `../download-installer_1.09ub untu2.1_all.udeb'.
 dpkg-genchanges
dpkg-genchanges: warning: unknown information field `Xb-Installer-Menu-Item' in input data in package's section of control info file
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: source only upload: Debian-native package
Now signing changes and any dsc files...
 signfile net-retriever_1.09ubuntu2.1.dsc darkness <darkness@caliginous.net>

You need a passphrase to unlock the secret key for
user: "darkness <darkness@caliginous.net>"
1024-bit DSA key, ID 7D9B4A32, created 2007-02-08


 signfile net-retriever_1.09ubuntu2.1_i386.changes darkness <darkness@caliginous .net>

You need a passphrase to unlock the secret key for
user: "darkness <darkness@caliginous.net>"
1024-bit DSA key, ID 7D9B4A32, created 2007-02-08


Successfully signed dsc and changes files

A few warnings there but (I think) they’re safe to ignore.

Making a cute local repository

We’re going to use two tools for this: dput and mini-dinstall. mini-dinstall helps you to maintain a local repository. dput’s job is to “upload” (read: copy) all the files you need into a place where mini-dinstall will process them. dput’s job may sound rather simple, but mind you that a single package has to have several files “uploaded” for it to be added into a repository: there’s a changes file, the .dsc file, the sources, maybe the Debian/Ubuntu-specific patch, and finally the resulting binary packages. dput makes this simple: hand it the changes file and it copies everything mini-dinstall will need.

$ pwd
/home/darkness/dpkg-build/net-retriever
$ # Note that I've changed directory from the last section.
$ sudo apt-get install mini-dinstall dput
[...]
$ mkdir ~/dpkg-build/repository
$ cat >~/.dput.cf
[DEFAULT]
default_host_main = local

[local]
fqdn = private
incoming = /home/darkness/dpkg-build/repository/mini-dinstall/incoming
method = local
run_dinstall = 0
post_upload_command = mini-dinstall -bv
allow_unsigned_uploads = 0
$ cat >~/.mini-dinstall.conf
[DEFAULT]
archivedir = /home/darkness/dpkg-build/repository
archive_style = flat
verify_sigs = 1
extra_keyrings = /etc/apt/trusted.gpg
generate_release = 1
release_signscript = /home/darkness/dpkg-build/repository/mini-dinstall/sign-release.sh

[dapper]
release_codename = dapper
$ # I run mini-dinstall -b just to make it create its directory hierarchy.
$ mini-dinstall -b
mini-dinstall [-1225434208] ERROR: Can't execute script "/home/darkness/dpkg-build/repository/mini-dinstall/sign-release.sh"
$ ls -1 ~/dpkg-build/repository/mini-dinstall
REJECT
incoming
mini-dinstall.log
$ cp /usr/share/doc/mini-dinstall/examples/sign-release.sh ~/dpkg-build/repository/mini-dinstall/
$ cat >~/dpkg-build/.gnupg/passphrase
your GPG key passphrase here

You should look at a few of the variables near the top of ~/dpkg-build/repository/mini-dinstall/sign-release.sh, GNUPGHOME and KEYID in particular. sign-release.sh wants to find the passphrase for your key in $GNUPGHOME/passphrase.

OK, now that all that setup is done, we can install our new net-retriever package into our new repository:

$ pwd
/home/darkness/dpkg-build/net-retriever
$ dput net-retriever_1.09ubuntu2.1_i386.changes
Upload package to host local
Checking Signature on .changes
gpg: Signature made Thu Feb  8 14:39:16 2007 EST using DSA key ID 7D9B4A32
gpg: Good signature from "darkness <darkness@caliginous.net>"
Good signature on /home/darkness/dpkg-build/net-retriever/net-retriever_1.09ubuntu2.1_i386.changes.
Checking Signature on .dsc
gpg: Signature made Thu Feb  8 14:39:14 2007 EST using DSA key ID 7D9B4A32
gpg: Good signature from "darkness <darkness@caliginous.net>"
Good signature on /home/darkness/dpkg-build/net-retriever/net-retriever_1.09ubuntu2.1.dsc.
Successfully uploaded packages.
Not running dinstall.
mini-dinstall [-1209952032] INFO: Booting mini-dinstall 0.6.21ubuntu1
mini-dinstall [-1209952032] INFO: Changing mode of "/home/darkness/dpkg-build/repository/mini-dinstall/incoming" to 750
mini-dinstall [-1209952032] INFO: Initializing archive indexer dapper
mini-dinstall [-1209952032] INFO: Initializing incoming processor
mini-dinstall [-1215902800] INFO: Created new installer thread (incoming)
mini-dinstall [-1215902800] INFO: Entering batch mode...
mini-dinstall [-1215902800] INFO: Examining "/home/darkness/dpkg-build/repository/mini-dinstall/incoming/net-retriever_1.09ubuntu2.1_i386.changes"
mini-dinstall [-1215902800] INFO: Preparing to install "/home/darkness/dpkg-build/repository/mini-dinstall/incoming/net-retriever_1.09ubuntu2.1_i386.changes" in archive dapper
mini-dinstall [-1215902800] INFO: Verifying signature on "/home/darkness/dpkg-build/repository/mini-dinstall/incoming/net-retriever_1.09ubuntu2.1_i386.changes"
mini-dinstall [-1215902800] INFO: Good signature on "/home/darkness/dpkg-build/repository/mini-dinstall/incoming/net-retriever_1.09ubuntu2.1_i386.changes"
mini-dinstall [-1215902800] INFO: Changing mode of "/home/darkness/dpkg-build/repository/mini-dinstall/incoming/net-retriever_1.09ubuntu2.1_i386.changes" to 600
mini-dinstall [-1215902800] INFO: Successfully installed net-retriever 1.09ubuntu2.1 to dapper
mini-dinstall [-1224389712] INFO: Created new thread (dapper) for archive indexer dapper
mini-dinstall [-1224389712] INFO: Entering batch mode...
mini-dinstall [-1224389712] INFO: Generating Packages file...
mini-dinstall [-1215902800] INFO: All packages in incoming dir installed; exiting
mini-dinstall [-1224389712] INFO: Packages generation complete
mini-dinstall [-1224389712] INFO: Generating Sources file...
mini-dinstall [-1224389712] INFO: Sources generation complete
mini-dinstall [-1224389712] INFO: Generating Release...
mini-dinstall [-1224389712] INFO: Running "/home/darkness/dpkg-build/repository/mini-dinstall/sign-release.sh /home/darkness/dpkg-build/repository/dapper/Release.dinstall-new"
mini-dinstall [-1224389712] INFO: Release generation complete
mini-dinstall [-1209952032] INFO: main thread exiting...
mini-dinstall [-1224389712] INFO: Thread "dapper" exiting

That’s a lot of output, but I wanted to include it so you could see what a successful run should look like. You can change -bv to -b in ~/.dput.cf but I find it useful to make sure my package has been installed. Here’s what my repository looks like now:

$ cd ~/dpkg-build/repository
$ ls -1
dapper
mini-dinstall
$ cd dapper
$ ls -1
Packages
Packages.gz
Release
Release.gpg
Sources
Sources.gz
download-installer_1.09ubuntu2.1_all.udeb
net-retriever_1.09ubuntu2.1.dsc
net-retriever_1.09ubuntu2.1.tar.gz
net-retriever_1.09ubuntu2.1_all.udeb
net-retriever_1.09ubuntu2.1_i386.changes

Wasn’t that easy?

Making the netboot image (finally)

Grab the debian-installer sources:

$ pwd
/home/darkness/dpkg-build
$ mkdir debian-installer
$ cd debian-installer/
$ apt-get source debian-installer
Reading package lists... Done
Building dependency tree... Done
Need to get 1165kB of source archives.
Get:1 http://us.archive.ubuntu.com dapper-updates/main debian-installer 20051026ubuntu36.6 (dsc) [1758B]
Get:2 http://us.archive.ubuntu.com dapper-updates/main debian-installer 20051026ubuntu36.6 (tar) [1163kB]
Fetched 1165kB in 0s (7983kB/s)
dpkg-source: extracting debian-installer in debian-installer-20051026ubuntu36.6
dpkg-source: unpacking debian-installer_20051026ubuntu36.6.tar.gz
$ cd debian-installer-20051026ubuntu36.6/build

There’s a README in this directory that gives you some idea of what to do. I’ll make a sources.list.udeb: the build scripts will make one for you, based on your /etc/apt/sources.list, but it doesn’t get it exactly right all the time (I see duplicated dapper-security sources). Also, I want to include dapper-proposed as well as the repository I just created.

$ cat >sources.list.udeb
deb http://us.archive.ubuntu.com/ubuntu dapper main/debian-installer
deb http://us.archive.ubuntu.com/ubuntu dapper-updates main/debian-installer
deb http://us.archive.ubuntu.com/ubuntu dapper-proposed main/debian-installer
deb http://security.ubuntu.com/ubuntu dapper-security main/debian-installer
deb copy:///home/darkness/dpkg-build/repository dapper/

The use of copy:// here is important for d-i to work; if you use file:// instead the packages don’t get downloaded where d-i can pick at them (i.e., presumably unpack them with udpkg). That trailing / on my local repository is important too.

Now we need to make two changes in some configuraton files. First, edit config/i386.cfg and change the kernel version to reflect the one you’re using from -proposed (e.g., KERNELVERSION = 2.6.15-50-386). Next, in pkg-lists/netboot/i386.cfg insert usb-storage-modules-${kernel:Version} on a new line below usb-modules-${kernel:Version}.

Before we can build the netboot image, we have to install any build dependencies for d-i.

$ sudo apt-get install libc6-pic
[...]
$ sudo apt-get build-dep debian-installer
[...]

libc6-pic is a build dependency of d-i; well, actually glibc-pic is, and libc6-pic provides glibc-pic. If you don’t specify libc6-pic explicitly apt-get complains.

OK, finally, lets build the netboot image: run fakeroot make build_netboot. (If you want, put http_proxy in the environment to specify a proxy server.) When this finishes successfully you should have dest/netboot/netboot.tar.gz which is just like the netboot.tar.gz that Ubuntu gives you, but with your updated packages.

Go forth and install Dapper server.

Notes after installation

My netboot image seemed to install pretty well. There were a few weird things:

  • The installer told grub to boot the first partition on the first disk, which was actually the Dell diagnostic partition. (/boot was the second partition.) Just a matter of changing hd(0,0) to hd(0,1).
  • Though I selected shadow passwords, they weren’t set up. pwconv fixed this, of course.

Otherwise the system seems fine.

I’ll note that I didn’t check for a newer kernel that could be used for installation in something like the dapper-updates distribution. That might have been a safer route than using -proposed.

No Comments

Leave A Comment

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS