Documentation

LMDB environments have various limits on the size and number of databases and concurrent readers.

The default limits are 1 MiB map size, 0 named databases (see Databases), and 126 concurrent readers. These can be adjusted freely, and in particular the mapSize may be set very large (limited only by available address space). However, LMDB is not optimized for a large number of named databases so maxDatabases should be kept to a minimum.

The default mapSize is intentionally small, and should be changed to something appropriate for your application. It ought to be a multiple of the OS page size, and should be chosen as large as possible to accommodate future growth of the database(s). Once set for an environment, this limit cannot be reduced to a value smaller than the space already consumed by the environment; however, it can later be increased.

If you are going to use any named databases then you will need to change maxDatabases to the number of named databases you plan to use. However, you do not need to change this field if you are only going to use the single main (unnamed) database.

Open an LMDB environment in either ReadWrite or ReadOnly mode. The FilePath argument may be either a directory or a regular file, but it must already exist; when creating a new environment, one should create an empty file or directory beforehand. If a regular file, an additional file with "-lock" appended to the name is automatically created for the reader lock table.

Note that an environment must have been opened in ReadWrite mode at least once before it can be opened in ReadOnly mode.

An environment opened in ReadOnly mode may still modify the reader lock table (except when the filesystem is read-only, in which case no locks are used).

To satisfy certain low-level LMDB requirements, please do not have opened the same environment (i.e., the same FilePath) more than once in the same process at the same time. Furthermore, please use the environment in the process that opened it (not after forking a new process).

Closes the given environment.

If you have merely a few dozen environments at most, there should be no need for this. (It is a common practice with LMDB to create one’s environments once and reuse them for the remainder of the program’s execution.)

To satisfy certain low-level LMDB requirements:

• Before calling this function, please call closeDatabase on all databases in the environment.
• Before calling this function, close all cursors and commit/abort all transactions on the environment. To make sure this requirement is satisified for read-only transactions, either (a) call waitReaders or (b) pass precreated cursors/transactions to readLMDB and unsafeReadLMDB.
• After calling this function, do not use the environment or any related databases, transactions, and cursors.

Gets a database with the given name.

If only one database is desired within the environment, the name can be Nothing (known as the “unnamed database”).

If one or more named databases (a database with a Just name) are desired, the maxDatabases of the environment’s limits should have been adjusted accordingly. The unnamed database will in this case contain the names of the named databases as keys, which one is allowed to read but not write.

Warning: When getting a named database for the first time (i.e., creating it), one must do so in the ReadWrite environment mode. (This restriction does not apply for the unnamed database.) In this case, this function spawns a bound thread and creates a temporary read-write transaction under the hood; see Transactions.

Closes the given database.

If you have merely a few dozen databases at most, there should be no need for this. (It is a common practice with LMDB to create one’s databases once and reuse them for the remainder of the program’s execution.)

To satisfy certain low-level LMDB requirements:

• Before calling this function, please make sure all read-write transactions that have modified the database have already been committed or aborted.
• After calling this function, do not use the database or any of its cursors again. To make sure this requirement is satisfied for cursors on read-only transactions, either (a) call waitReaders or (b) pass precreated cursors/transactions to readLMDB and unsafeReadLMDB.

A type for an optional thing where we want to fix the transaction mode to ReadOnly in the nothing case. (Maybe isn’t powerful enough for this.)

A type for an Either-like choice where we want to fix the transaction mode to ReadOnly in the Left case. (Either isn’t powerful enough for this.)

Use unsafe FFI calls under the hood. This can increase iteration speed, but one should bear in mind that unsafe FFI calls, since they block all other threads, can have an adverse impact on the performance of the rest of the program.

Internal.

Creates an unfold with which we can stream key-value pairs from the given database.

If an existing transaction and cursor are not provided, there are two possibilities: (a) If a chunk size is not provided, a read-only transaction and cursor are automatically created for the entire duration of the unfold. (b) Otherwise, new transactions and cursors are automatically created according to the desired chunk size. In this case, each transaction (apart from the first one) starts as expected at the key next to (i.e., the largest/smallest key less/greater than) the previously encountered key.

If you want to iterate through a large database while avoiding a long-lived transaction (see Transactions), it is your responsibility to either chunk up your usage of readLMDB (with which readStart can help) or specify a chunk size as described above.

Runtime consideration: If you call readLMDB very frequently without a precreated transaction and cursor, you might find upon profiling that a significant time is being spent at mdb_txn_begin, or find yourself having to increase maxReaders in the environment’s limits because the transactions and cursors are not being garbage collected fast enough. In this case, please consider precreating a transaction and cursor.

If you don’t want the overhead of intermediate ByteStrings (on your way to your eventual data structures), use unsafeReadLMDB instead.

Similar to readLMDB, except that it has an extra UseUnsafeFFI parameter.

Internal.

Similar to readLMDB, except that the keys and values are not automatically converted into Haskell ByteStrings.

To ensure safety, please make sure that the memory pointed to by the CStringLen for each key/value mapping function call is (a) only read (and not written to); and (b) not used after the mapping function has returned. One way to transform the CStringLens to your desired data structures is to use unsafePackCStringLen.

Similar to unsafeReadLMDB, except that it has an extra UseUnsafeFFI parameter.

Internal.

State that stays fixed in readLMDB iterations.

Internal.

Looks up the value for the given key in the given database.

If an existing transaction is not provided, a read-only transaction is automatically created internally.

Runtime consideration: If you call getLMDB very frequently without a precreated transaction, you might find upon profiling that a significant time is being spent at mdb_txn_begin, or find yourself having to increase maxReaders in the environment’s limits because the transactions are not being garbage collected fast enough. In this case, please consider precreating a transaction.

A read-only (tmode: ReadOnly) or read-write (tmode: ReadWrite) transaction.

emode: the environment’s mode. Note: ReadOnly environments can only have ReadOnly transactions; we enforce this at the type level.

Begins an LMDB read-only transaction on the given environment.

For read-only transactions returned from this function, it is your responsibility to (a) make sure the transaction only gets used by a single readLMDB, unsafeReadLMDB, or getLMDB at the same time, (b) use the transaction only on databases in the environment on which the transaction was begun, (c) make sure that those databases were already obtained before the transaction was begun, (d) dispose of the transaction with abortReadOnlyTransaction, and (e) be aware of the caveats regarding long-lived transactions; see Transactions.

To easily manage a read-only transaction’s lifecycle, we suggest using withReadOnlyTransaction.

Disposes of a read-only transaction created with beginReadOnlyTransaction.

It is your responsibility to not use the transaction or any of its cursors afterwards.

Creates a temporary read-only transaction on which the provided action is performed, after which the transaction gets aborted. The transaction also gets aborted upon exceptions.

You have the same responsibilities as documented for beginReadOnlyTransaction (apart from the transaction disposal).

A cursor.

Opens a cursor for use with readLMDB or unsafeReadLMDB. It is your responsibility to (a) make sure the cursor only gets used by a single readLMDB or unsafeReadLMDB at the same time, (b) make sure the provided database is within the environment on which the provided transaction was begun, and (c) dispose of the cursor with closeCursor.

To easily manage a cursor’s lifecycle, we suggest using withCursor.

Disposes of a cursor created with openCursor.

Creates a temporary cursor on which the provided action is performed, after which the cursor gets closed. The cursor also gets closed upon exceptions.

You have the same responsibilities as documented for openCursor (apart from the cursor disposal).

By default, we start reading from the beginning of the database (i.e., from the smallest key) and iterate in forward direction.

Direction of key iteration.

The key from which an iteration should start.

Begins an LMDB read-write transaction on the given environment.

Unlike read-only transactions, a given read-write transaction is not allowed to stray from the OS thread on which it was begun, and it is your responsibility to make sure of this. You can achieve this with, e.g., runInBoundThread.

Additionally, for read-write transactions returned from this function, it is your responsibility to (a) use the transaction only on databases in the environment on which the transaction was begun, (b) make sure that those databases were already obtained before the transaction was begun, (c) commit/abort the transaction with commitReadWriteTransaction/abortReadWriteTransaction, and (d) be aware of the caveats regarding long-lived transactions; see Transactions.

To easily manage a read-write transaction’s lifecycle, we suggest using withReadWriteTransaction.

Aborts a read-write transaction created with beginReadWriteTransaction.

It is your responsibility to not use the transaction afterwards.

Commits a read-write transaction created with beginReadWriteTransaction.

It is your responsibility to not use the transaction afterwards.

Spawns a new bound thread and creates a temporary read-write transaction on which the provided action is performed, after which the transaction gets committed. The transaction gets aborted upon exceptions.

You have the same responsibilities as documented for beginReadWriteTransaction (apart from running it on a bound thread and committing/aborting it).

• OverwriteAllow: When a key reoccurs, overwrite the value.
• OverwriteDisallow: When a key reoccurs, don’t overwrite and hand the maladaptive key-value pair to the accumulator.
• OverwriteAppend: Assume the input data is already increasing, which allows the use of MDB_APPEND under the hood and substantially improves write performance. Hand arriving key-value pairs in a maladaptive order to the accumulator.

A fold for (key, new value).

A fold for (key, new value, old value). This has the overhead of getting the old value.

A function that shows a database key.

A function that shows a database value.

Throws upon the first maladaptive key. If desired, shows the maladaptive key-value pair in the exception.

Throws upon the first maladaptive key where the old value differs from the new value. If desired, shows the maladaptive key-value pair with the old value in the exception.

Silently ignores maladaptive keys.

Gracefully stops upon the first maladaptive key.

By default, we allow overwriting.

A chunk size.

Chunks up the incoming stream of key-value pairs using the desired chunk size. One can try, e.g., ChunkBytes mebibyte (1 MiB chunks) and benchmark from there.

The chunks are processed using the desired fold.

Chunks up the incoming stream of key-value pairs using the desired chunk size. One can try, e.g., ChunkBytes mebibyte (1 MiB chunks) and benchmark from there.

Writes a chunk of key-value pairs to the given database. Under the hood, it uses writeLMDB surrounded with a withReadWriteTransaction.

Similar to writeLMDBChunk, except that it has an extra UseUnsafeFFI parameter.

Internal.

Creates a fold that writes a stream of key-value pairs to the provided database using the provided transaction.

If you have a long stream of key-value pairs that you want to write to an LMDB database while avoiding a long-lived transaction (see Transactions), you can use the functions for chunked writing.

Similar to writeLMDB, except that it has an extra UseUnsafeFFI parameter.

Internal.

Waits for active read-only transactions on the given environment to finish. Note: This triggers garbage collection.

Clears, i.e., removes all key-value pairs from, the given database.

Warning: Under the hood, this function spawns a bound thread and creates a potentially long-lived read-write transaction; see Transactions.

Deletes the given key from the given database using the given transaction.

By default, we do not assume the key being deleted already exists in the database.

A convenience constant for obtaining 1 KiB.

A convenience constant for obtaining 1 MiB.

A convenience constant for obtaining 1 GiB.

A convenience constant for obtaining 1 TiB.

A type class for ReadOnly and ReadWrite environments and transactions.

Enforces at the type level that ReadWrite environments support both ReadWrite and ReadOnly transactions, but ReadOnly environments support only ReadOnly transactions.

Keeps track of the ThreadId of the current read-write transaction.

Utility function for getting a named database with a read-only transaction, returning Nothing if it was not found.

Internal.

A utility function for detecting a few user errors.

Internal.

liftedBracket2 acquire failure success thing

Same as Control.Exception.Lifted.bracket (from lifted-base) except it distinguishes between failure and success.

Notes:

• When acquire, success, or failure throw exceptions, any monadic side effects in m will be discarded.
• When thing throws an exception, any monadic side effects in m produced by thing will be discarded, but the side effects of acquire and (non-excepting) failure will be retained.
• When (following a thing success) success throws an exception, any monadic side effects in m produced by success will be discarded, but the side effects of acquire, thing, and (non-excepting) failure will be retained.

Internal.

Same as Control.Exception.bracket except it distinguishes between failure and success. (If the success action throws an exception, the failure action gets called.)

Internal.