OpenLDAP mdb backend ready for production

OpenLDAP's new memory-mapped backend, mdb, is ready for production. It was first introducted over a year ago, and since then, the OpenLDAP developers (Howard Chu in particular) have put quite a bit of work into getting all of the bugs worked out.

The OpenLDAP mailing lists report that several companies are deploying it in production:
http://www.openldap.org/lists/openldap-technical/201208/msg00137.html

The latest version of email/collaboration software Zimbra uses the mdb backend as well:
http://wiki.zimbra.com/wiki/OpenLDAP_Tuning_Keys_8.0#OpenLDAP_Tuning_Keys

All of this leads me to believe it's production-ready. I've been running it myself at my own job for a couple weeks now, and it's actually lived up to the hype. It really is faster, uses less memory, and is way easier to configure. You can read more about the advantages of mdb here: Coming soon: memory-mapped db for OpenLDAP

Here's a short list of what I would consider the caveats:

• Since it's new technology, I highly recommend compiling the latest version of OpenLDAP (as of this posting 2.4.33) to get the latest bug fixes and performance improvements. I have a page of useful tips for compiling OpenLDAP to get the most out of it here: compiling OpenLDAP.


• Write performance is still being optimized. My personal recommendation would be to continue to use hdb on the master server (properly optimized), and mdb on the replica servers.


• I'm guessing because of the previous point, it can take quite a bit longer to do the initial database load. Using hdb it took a few hours to slapadd our database, but it's taking around 12 hours to slapadd it using mdb. It's worth the wait, though.
  (see comments)

I've updated my wiki on optimizing OpenLDAP with instructions for mdb. Instead of worrying about locks and three different levels of caches, there's just one important option which is trivial to configure:
Optimizing OpenLDAP

More information about mdb is available here:
http://highlandsun.com/hyc/mdb/

Now, if Symas would only post some updated benchmarks, I could point to just how much faster OpenLDAP is than the competition...

coming soon: memory-mapped db for OpenLDAP

short version:

I just ran across a site with news of a new database backend for OpenLDAP that's designed to be completely mapped to memory and is supposed to be faster, more memory efficient, and much easier to configure:

Memory-mapped Database for OpenLDAP

long version:

I'm a huge fan of OpenLDAP. not only is it the fastest and most scalable implementation of LDAP (see at the bottom for sources), best of all it's open-source. configuring it for optimal performance, however, is easier said than done. you have to configure indexes, the database cache size, the IDL cache size and of course the good ole entry cache size. Howard Chu, the current chief architect of OpenLDAP, describes the process pretty well:

"...it requires careful tuning to get good results and the tuning aspects can be quite complex. Data comes through three separate layers of caches before it may be used, and each cache layer has a significant footprint. Balancing the three layers against each other can be a difficult juggling act."

the good news: that quote comes from a site I just ran across where Chu announces a new database backend that's designed to be completely mapped to memory, known as "back-mdb". in Chu's description of back-mdb, he uses words like "extremely fast," "memory efficiency," and my favorite, "trivial configuration." I can't wait! (yes, I'm a giant nerd)

Chu said back-mdb will be ready sometime this month at the earliest, but whenever it comes, it will be worth waiting for. here's the link if you want to check out the details:

Memory-mapped Database for OpenLDAP

and here are those sources I promised you, ripped entirely from my wiki:

• fastest implementation of LDAP:
  
  
  
  • compared to 389/Fedora DS
  
  
  • compared to OpenDS and ApacheDS
  
  
  • compared to Oracle Internet Directory and OpenDS
  
  
  • compared to AD
  
  
  
  


• most scalable implementation of LDAP

libgssapiv2.so.2: undefined symbol: gss_nt_service_name

More compiling woes...

This time I was compiling cyrus-sasl 2.1.23 for use with OpenLDAP. I got it all compiled, and tried to run and ldapsearch using GSSAPI, but I got this error:
$ ldapsearch -b dc=example,dc=com -LLL
ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
additional info: SASL(-4): no mechanism available: No worthy mechs found

When I looked at /var/log/messages, I saw this:
Apr 6 16:42:01 ldap ldapsearch: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.2: /usr/local/lib/sasl2/libgssapiv2.so.2: undefined symbol: gss_nt_service_name
Apr 6 16:42:01 ldap ldapsearch: No worthy mechs found

I realized that for some reason the SASL GSSAPI plugin wasn't compiling, which I noticed when I ran /usr/local/sbin/pluginviewer -s (Plugin "gssapiv2" [loaded] wasn't in the list)

After googling, I found a possible solution here:
http://lists.freebsd.org/pipermail/freebsd-questions/2011-January/226495.html

But I didn't like it because it seemed a little hackish. then I ran across a patch posted here:
http://lists.freebsd.org/pipermail/freebsd-questions/2011-February/227044.html

It seemed like a much better solution, but there seemed to be quite a bit in the patch that wasn't relevant to my problem, so I trimmed it down to two changes:

--- configure.dist 2011-04-06 10:30:07.000000000 -0500
+++ configure 2011-04-06 11:25:17.000000000 -0500
@@ -11122 +11122 @@
- if test "$gss_impl" = "cybersafe" -o "$gss_impl" = "cybersafe03"; then
+ if test "$gss_impl" = "cybersafe" -o "$gss_impl" = "cybersafe03" -o "$gss_impl" = "heimdal"; then
@@ -11193 +11193 @@
- if test "$gss_impl" = "cybersafe" -o "$gss_impl" = "cybersafe03"; then
+ if test "$gss_impl" = "cybersafe" -o "$gss_impl" = "cybersafe03" -o "$gss_impl" = "heimdal"; then

You can download the raw patch here:
https://gist.github.com/bmaupin/2a4d08edc128d785ad46

heimdal 1.4 missing lib/otp/version-script.map

I was compiling Heimdal kerberos 1.4 on RHEL 5.6 and got the following error:

/usr/bin/ld: cannot open linker script file ./version-script.map: No such file or directory

After some poking around, it appears that lib/otp/version-script.map is missing from the 1.4 source, which is weird since I downloaded it right from the big "Download 1.4" link from their website (http://www.h5l.org/), which links to http://www.h5l.org/dist/src/heimdal-1.4.tar.gz.

So, to fix the problem, you can download the missing file right from their git repo, here: https://github.com/heimdal/heimdal/raw/e3044663df2cfcafd5bf4e2ea6f2a1ba1503d8ea/lib/otp/version-script.map

Just place it in heimdal-1.4/lib/otp, and you should be good to go.

So the whole process would look something like this:

wget http://www.h5l.org/dist/src/heimdal-1.4.tar.gz
tar xvf heimdal-1.4.tar.gz
wget --no-check-certificate https://github.com/heimdal/heimdal/raw/e3044663df2cfcafd5bf4e2ea6f2a1ba1503d8ea/lib/otp/version-script.map -P heimdal-1.4/lib/otp

...and then you can go on with the configure, make, etc. to build it.

Edit:
Of course if you don't care about OTP (one time password) support, you can just disable this altogether when compiling and not worry about the missing file. Just add the "--disable-otp" flag when running the configure command:

./configure --disable-otp

Heimdal 1.3: BAD_ENCRYPTION_TYPE

Solution:

Edit the Kerberos configuration file (for me it's /etc/krb5.conf), and add this line in the [libdefaults] section:

allow_weak_crypto = true

Details:

More of the same... (see my posts on Berkeley DB) I compiled the latest version of Heimdal Kerberos the other day (it's the recommended flavor of Kerberos to use with OpenLDAP), and after getting everything set up, I would get these errors when trying to run LDAP searches:

SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:  Miscellaneous failure (see text) (encryption type 2 not supportedt))

(sometimes it also said encryption type 1 not supportedt, again with the t on the end of supported)

My first thought was maybe I didn't have a valid Kerberos ticket, but when running kinit, I would get this:

kinit: krb5_get_init_creds: BAD_ENCRYPTION_TYPE"

Google searches on those terms didn't get me far, but after looking through the Heimdal documentation, I realized they deprecated support for DES encryption in Heimdal 1.3. The solution I discovered for enabling it again is posted above.

Et voila! Back in business.

More information: DES will die in Heimdal 1.3

Compiling Berkeley DB on GNU/Linux

When compiling Berkeley DB on GNU/Linux (in particular if it's being compiled to be used with OpenLDAP), make sure you compile it with the following flags:

--enable-posixmutexes --with-mutex=POSIX/pthreads

I happened to run across that buried in the openldap-software mailing list archives. Here are some more details:

"On x86 though, they have a hybrid mutex mechanism which is enabled by default. It uses both assembly language spinlocks and pthread mutexes, first spinning in the assembly language lock some number of times before falling back onto the mutex. They claimed that this improved performance in their tests, because pthread mutexes can be very expensive on some platforms. In my tests on x86_64 Linux however, it just forced CPU usage to 100% (200% actually, dualcore) and slowed down overall throughput.

You can give an explicit "--with-mutex=POSIX/pthread" argument when configuring BerkeleyDB to avoid the hybrid mutex scheme"

http://www.openldap.org/lists/openldap-software/200708/msg00344.html

"BerkeleyDB will use excessive CPU on Linux unless you configure --with-mutex=POSIX/pthreads. By default it uses "hybrid" mutexes, which use an assembly language spinlock loop before falling back to pthread mutex. The spinlock will chew up all of your CPU on multiprocessor machines..."

http://www.openldap.org/lists/openldap-devel/200809/msg00085.html

More:
http://www.openldap.org/lists/openldap-devel/200902/msg00064.html
http://www.openldap.org/lists/openldap-technical/200906/msg00015.html