streamly-lmdb-0.3.0: Stream data to or from LMDB databases using the streamly library.
Safe HaskellNone
LanguageHaskell2010

Streamly.External.LMDB

Description

The functionality for the limits and getting the environment and database were mostly obtained from the lmdb-simple library.

Synopsis

Types

data Database mode Source #

data Environment mode Source #

data Limits Source #

LMDB environments have various limits on the size and number of databases and concurrent readers.

Constructors

Limits 

Fields

  • mapSize :: !Int

    Memory map size, in bytes (also the maximum size of all databases).

  • maxDatabases :: !Int

    Maximum number of named databases.

  • maxReaders :: !Int

    Maximum number of concurrent ReadOnly transactions (also the number of slots in the lock table).

class Mode a Source #

Minimal complete definition

isReadOnlyMode

Instances

Instances details
Mode ReadOnly Source # 
Instance details

Defined in Streamly.External.LMDB.Internal

Mode ReadWrite Source # 
Instance details

Defined in Streamly.External.LMDB.Internal

data ReadWrite Source #

Instances

Instances details
Mode ReadWrite Source # 
Instance details

Defined in Streamly.External.LMDB.Internal

data ReadOnly Source #

Instances

Instances details
Mode ReadOnly Source # 
Instance details

Defined in Streamly.External.LMDB.Internal

data ReadDirection Source #

Direction of key iteration.

Constructors

Forward 
Backward 

Instances

Instances details
Show ReadDirection Source # 
Instance details

Defined in Streamly.External.LMDB

data ReadOptions Source #

Constructors

ReadOptions 

Fields

Instances

Instances details
Show ReadOptions Source # 
Instance details

Defined in Streamly.External.LMDB

Environment and database

defaultLimits :: Limits Source #

The default limits are 1 MiB map size, 0 named databases, and 126 concurrent readers. These can be adjusted freely, and in particular the mapSize may be set very large (limited only by available address space). However, LMDB is not optimized for a large number of named databases so maxDatabases should be kept to a minimum.

The default mapSize is intentionally small, and should be changed to something appropriate for your application. It ought to be a multiple of the OS page size, and should be chosen as large as possible to accommodate future growth of the database(s). Once set for an environment, this limit cannot be reduced to a value smaller than the space already consumed by the environment, however it can later be increased.

If you are going to use any named databases then you will need to change maxDatabases to the number of named databases you plan to use. However, you do not need to change this field if you are only going to use the single main (unnamed) database.

openEnvironment :: Mode mode => FilePath -> Limits -> IO (Environment mode) Source #

Open an LMDB environment in either ReadWrite or ReadOnly mode. The FilePath argument may be either a directory or a regular file, but it must already exist. If a regular file, an additional file with "-lock" appended to the name is used for the reader lock table.

Note that an environment must have been opened in ReadWrite mode at least once before it can be opened in ReadOnly mode.

An environment opened in ReadOnly mode may still modify the reader lock table (except when the filesystem is read-only, in which case no locks are used).

getDatabase :: Mode mode => Environment mode -> Maybe String -> IO (Database mode) Source #

Utility

clearDatabase :: Mode mode => Database mode -> IO () Source #

Clears, i.e., removes all key-value pairs from, the given database.

Reading

readLMDB :: (MonadIO m, Mode mode) => Database mode -> ReadOptions -> Unfold m Void (ByteString, ByteString) Source #

Creates an unfold with which we can stream key-value pairs from the given database.

A read transaction is kept open for the duration of the unfold; one should therefore bear in mind LMDB's caveats regarding long-lived transactions.

If you don’t want the overhead of intermediate ByteStrings (on your way to your eventual data structures), use unsafeReadLMDB instead.

unsafeReadLMDB :: (MonadIO m, Mode mode) => Database mode -> ReadOptions -> (CStringLen -> IO k) -> (CStringLen -> IO v) -> Unfold m Void (k, v) Source #

Creates an unfold with which we can stream key-value pairs from the given database.

A read transaction is kept open for the duration of the unfold; one should therefore bear in mind LMDB's caveats regarding long-lived transactions.

To ensure safety, make sure that the memory pointed to by the CStringLen for each key/value mapping function call is (a) only read (and not written to); and (b) not used after the mapping function has returned. One way to transform the CStringLens to your desired data structures is to use unsafePackCStringLen.

Writing

writeLMDB :: MonadIO m => Database ReadWrite -> WriteOptions -> Fold m (ByteString, ByteString) () Source #

Creates a fold with which we can stream key-value pairs into the given database.

It is the responsibility of the user to execute the fold on a bound thread.

The fold currently cannot be used with a scan. (The plan is for this shortcoming to be remedied with or after a future release of streamly that addresses the underlying issue.)

Please specify a suitable transaction size in the write options; the default of 1 (one write transaction for each key-value pair) could yield suboptimal performance. One could try, e.g., 100 KB chunks and benchmark from there.