MongoDB: Avoid the "majority" write concern when using arbiters

Here at Monetology we use MongoDB for most of our storage needs.  I'm not at all anti-relational and in fact I tend to be suspicious of a lot of the claims that relational DBs "don't scale to today's data" or "weren't designed for the web."  NoSQL isn't a magic pill that suddenly imparts awesome scalability.

Nonetheless, MongoDB has a lot of nice properties that make it a good choice for a variety of applications.  In particular, its easy to understand much of what its doing.  If you have a decent background in distributed systems (I do) and file systems (I don't, but I'm working on that) then MongoDB just make sense much of the time.

Of course, this doesn't stop me from screwing things up...  A few nights ago was one of those times.

Recently I've been familiarizing myself with replica sets, and I tried out the "majority" write concern.  Things worked fine when all mongods were up, but as soon as I would kill one our app would lock up; a new primary would get elected (if necessary) but all writes would hang.  After a lot of banging my head against a wall I realized why: I was only using 2 full replicas and 1 arbiter.  After killing a full replica, although I still had a majority of nodes available (2 out of 3), that's not what the majority write concern means.  Instead, it requires that your write is received by a majority of replicas.  Since I only had 1 full replica still available to receive writes, all writes were blocking as they waited for a second replica to appear.

There are two obvious solutions: either upgrade the arbiter to a full replica, or reduce the write concern from "majority" to "1".  Although one could conceive of some scenario where a majority write concern wouldn't be totally pointless with a replica set containing an arbiter (e.g. maybe a 5-node replica set with 4 full replicas and 1 arbiter), it seems to me like this makes for a slightly confusing combination and I'd advise avoiding it.