darkness

The problem:

You’ve got PAM configured and working for authentication against an LDAP server, and now you want VMware Server to authenticate users in the same way. You’re using SSL or TLS to talk to your LDAP server. I say “Fedora Core 5,” but that just happens to be what I have installed; I think this will apply to any system that has OpenSSL != 0.9.7 (FC5 has 0.9.8, and I think at least SuSE 10.1). When you edit /etc/pam.d/vmware-authd to look to your LDAP server for authentication, just like the rest of your system, you get things like:

Jul 28 15:00:07 verin /usr/sbin/vmware-authd[13560]: pam_ldap: ldap_starttls_s: Connect error

Apparently, the problem is that VMware Server comes with OpenSSL 0.9.7, but your system’s LDAP libraries are linked with OpenSSL 0.9.8. And that, apparently, doesn’t work.

My solution:

Lets make our own LDAP libraries linked against OpenSSL 0.9.7.

First, make a directory where we “install” all of these libraries to, temporarily:

tmp $ mkdir root

Next we need OpenSSL 0.9.7. Duh.

tmp $ tar -zxf openssl-0.9.7m.tar.gz
tmp $ cd openssl-0.9.7m
openssl-0.9.7m $ ./config --shared --prefix=~/tmp/root/usr
[... OpenSSL configures ...]
openssl-0.9.7m $ MAKEFLAGS=-j8 make && make install
[... OpenSSL compiles and installs ...]
openssl-0.9.7m $ cd ..
tmp $ ls -al root/usr/lib/libssl.so
lrwxrwxrwx 1 darkness darkness 11 Jul 28 17:55 root/usr/lib/libssl.so -> libssl.so.0

I use ~/tmp/root/usr here instead of just ~/tmp/root because OpenLDAP will later be very unhappy without going into a usr dir, and this way I don’t have two separate directories to put in variables like LD_LIBRARY_PATH later.

Now OpenLDAP, linked with our freshly-built OpenSSL 0.9.7:

tmp $ tar -zxf openldap-2.3.37.tgz
tmp $ cd openldap-2.3.37
openldap-2.3.37 $ CPPFLAGS=-I$HOME/tmp/root/usr/include \
                  LDFLAGS=-L$HOME/tmp/root/usr/lib \
                  LD_LIBRARY_PATH=$HOME/tmp/root/usr/lib \
                  ./configure --prefix=/usr --sysconfdir=/etc \
                  --datadir=/usr/share --localstatedir=/var \
                  --disable-slapd --disable-slurpd --disable-backends \
                  --disable-overlays --with-threads=posix \
                  --disable-static -enable-shared --enable-dynamic \
                  --enable-local --with-tls
[... OpenLDAP configures ...]
openldap-2.3.37 $ make depend && MAKEFLAGS=-j8 make
[... OpenLDAP builds ...]
openldap-2.3.37 $ make install DESTDIR=$HOME/tmp/root
[... OpenLDAP installs ...]
openldap-2.3.37 $ cd ..
tmp $ ls -al root/usr/lib/libldap.so
lrwxrwxrwx 1 darkness darkness 21 Jul 28 18:01 root/usr/lib/libldap.so -> libldap-2.3.so.0.2.25

I’m not sure that I didn’t build too much with those flags, but they worked. Also note that if you don’t specify paths to configure (I may have specified one or two that aren’t strictly necessary for us) you will be unhappy; i.e., OpenLDAP won’t know where to look for ldap.conf.

Now to install:

tmp $ cd root
root $ sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/libldap-2.3.so.0
root $ sudo install -o 0 -g 0 -m 444 usr/lib/libldap-2.3.so.0.2.25 \
                    /usr/lib/vmware/lib/libldap-2.3.so.0/
root $ sudo ln -s libldap-2.3.so.0.2.25 \
                  /usr/lib/vmware/lib/libldap-2.3.so.0/libldap-2.3.so.0
root $ sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/liblber-2.3.so.0
root $ sudo install -o 0 -g 0 -m 444 usr/lib/liblber-2.3.so.0.2.25 \
                    /usr/lib/vmware/lib/liblber-2.3.so.0/
root $ sudo ln -s liblber-2.3.so.0.2.25 \
                  /usr/lib/vmware/lib/liblber-2.3.so.0/liblber-2.3.so.0

Finally, you need to edit /etc/xinetd.d/vmware-authd to set LD_LIBRARY_PATH. Mine ends up looking like:

# default: on
# description: The VMware remote access authentification daemon
service vmware-authd
{
    disable         = no
    port            = 902
    socket_type     = stream
    protocol        = tcp
    wait            = no
    user            = root
    server          = /usr/sbin/vmware-authd
    type            = unlisted
    env             = LD_LIBRARY_PATH=/usr/lib/vmware/lib/libldap-2.3.so.0:/usr/lib/vmware/lib/liblber-2.3.so.0
}

(In reality I had to do all of the above several times while writing this, so I hope everything here represents what I really did. If it doesn’t work, you can try looking at my vmware-build-pam_ldap.sh script that I actually used to reproduce this build, but read it first since it might do something you don’t like involving rm -rf. Also, I could have just put all the LDAP libraries in something like /usr/lib/vmware/lib/local-openldap rather than two separate directories; that would make the LD_LIBRARY_PATH in the above file shorter.)

Note that pam_ldap doesn’t need to be rebuilt, at least not on my system. I think this is the case because the soname on the libraries that ship with VMware Server match the soname on the libraries that we built above.

When the above was done, I was able to log in to VMware Server as a user in LDAP. The above method has the benefit of only giving new libraries to VMware, not anything else on the system. (Some people had suggested compiling and installing OpenSSL 0.9.7 and OpenLDAP system-wide, which I hate doing.)