lookupd

Imagine the following:

• You can specify alpha keys (case sensitive/insensitive)
• You can specify numeric keys (2^64 range)
• Good for large on-line systems, with more users than you can handle comfortably (memory, disk, network, cache)
• The "normal" (or operating) state, is when all keys and values are in memory (sort of like in-memory database)
• The "degraded" state, is when some values (not keys) are on disk, in LRU (least recently used) basis
• As soon as you plug another node in, the system will re-balance, and you'll get back to normal (non-degraded state)
• Nodes can be defined for QUERY or PERSISTENCE (or both, though not ideal)
• No database, filer, storage back-end needed... make best use of "spindles"
• Supports both TCP and UDP for query interface, outrageous performance!
• Makes full use of Dual core 2 Duos... or any number of threads/cpu's you throw at it... optimized for 2-8 cores per node
• 64 bit app (compiles for 32 for proof of concept)
• Uses all the RAM you can throw at it... 8GB, 16GB, 32GB... no RAM disk, but real usage

Good uses?

• Millions of rows/records, need to keep a BLOb quickly accessible
  • Have you ever actually tried putting 50 million files/blobs on a filer, or a file system?
• Complex session data caching
  • When you have hundreds of thousands of queries per second, from hundreds of web servers...
• Ran out of database scale-out capacity
• Replication/scalability problems
• Need a lookup (i.e. zero to one record returned, not a result set)
• Hundreds of apache servers pounding on master/slave database system

This is not a database... if you think a database is what you need for your user lookups, then go on with your life — come back when you figure it out.

This project will expose a database like API, to lookups, but optimized for WEB 2.0 type uses.

Close cousins:

• memcached
  • except, lookupd is not a cache — it's a storage system
  • a cache, you'd have to warm up, and you would not use it for storage (persistence)
• Google File System
  • except, lookupd is about providing multiple keys, all memory resident
• MySQL scaleout
  • lookupd scales both horizontally and vertically
  • lookupd does not have SQL query language (at least yet)
  • lookupd is a multi-master, keys'n'blobs lookup system, that distributes and replicates automatiically
• Berkeley DataBase
  • BDB is really nice for lookup systems, but with all the flexibility you lose some of the control
  • lookupd is strictly about scaling and no compromises

Differentiation:

• Scaleable — add nodes, automatic balancing/distribution of namespace
• Multiple protocols — TCP and UDP — think of named...
• Lookups, not queries — single results, not result sets
• Persisted — uses file storage (DASD, NAS, SAN) but in single-file style, to avoid name space and caching issues
• Distributed — by masks on primary keys, to provide scalability on both horizontal scale-out, as well as persistence
• Replication — automatically replicates both to horizontal distribution, as well as persisted distribution

Code / downloads:

• Will follow shortly... code is alpha quality right now (4/30/07)
• Initial platform: POSIX (Linux 2.6), IPv4, pthreads, GNU C++
• Client interfaces: Wire protocol, Perl, Java
• Get in touch with me, if you want to do alpha testing or contribute...
• License: GPL

Status:

• 4/30/07 — Design almost complete