Safe Haskell | None |
---|---|
Language | Haskell2010 |
- mkEsScribe :: EsScribeCfg -> BHEnv -> IndexName -> MappingName -> Severity -> Verbosity -> IO (Scribe, IO ())
- data EsScribeSetupError
- = CouldNotCreateIndex !Reply
- | CouldNotCreateMapping !Reply
- data EsQueueSize
- mkEsQueueSize :: Int -> Maybe EsQueueSize
- data EsPoolSize
- mkEsPoolSize :: Int -> Maybe EsPoolSize
- data EsScribeCfg
- essRetryPolicy :: EsScribeCfg -> RetryPolicy
- essQueueSize :: EsScribeCfg -> EsQueueSize
- essPoolSize :: EsScribeCfg -> EsPoolSize
- essAnnotateTypes :: EsScribeCfg -> Bool
- essIndexSettings :: EsScribeCfg -> IndexSettings
- essIndexSharding :: EsScribeCfg -> IndexShardingPolicy
- data IndexShardingPolicy
- = NoIndexSharding
- | MonthlyIndexSharding
- | WeeklyIndexSharding
- | DailyIndexSharding
- | HourlyIndexSharding
- | EveryMinuteIndexSharding
- | CustomIndexSharding (forall a. Item a -> [IndexNameSegment])
- newtype IndexNameSegment = IndexNameSegment {
- indexNameSegment :: Text
- defaultEsScribeCfg :: EsScribeCfg
- mkDocId :: IO DocId
- module Katip.Scribes.ElasticSearch.Annotations
- roundToSunday :: Day -> Day
Building a scribe
:: EsScribeCfg | |
-> BHEnv | |
-> IndexName | Treated as a prefix if index sharding is enabled |
-> MappingName | |
-> Severity | |
-> Verbosity | |
-> IO (Scribe, IO ()) | Returns a finalizer that will gracefully flush all remaining logs before shutting down workers |
Scribe configuration
data EsScribeSetupError Source
CouldNotCreateIndex !Reply | |
CouldNotCreateMapping !Reply |
mkEsQueueSize :: Int -> Maybe EsQueueSize Source
mkEsPoolSize :: Int -> Maybe EsPoolSize Source
data EsScribeCfg Source
essRetryPolicy :: EsScribeCfg -> RetryPolicy Source
Retry policy when there are errors sending logs to the server
essQueueSize :: EsScribeCfg -> EsQueueSize Source
Maximum size of the bounded log queue
essPoolSize :: EsScribeCfg -> EsPoolSize Source
Worker pool size limit for sending data to the
essAnnotateTypes :: EsScribeCfg -> Bool Source
Different payload items coexist in the "data" attribute in ES. It is possible for different payloads to have different types for the same key, e.g. an "id" key that is sometimes a number and sometimes a string. If you're having ES do dynamic mapping, the first log item will set the type and any that don't conform will be *discarded*. If you set this to True, keys will recursively be appended with their ES core type. e.g. "id" would become "id::l" and "id::s" automatically, so they won't conflict. When this library exposes a querying API, we will try to make deserialization and querying transparently remove the type annotations if this is enabled.
essIndexSettings :: EsScribeCfg -> IndexSettings Source
data IndexShardingPolicy Source
How should katip store your log data?
- NoIndexSharding will store all logs in one index name. This is the simplest option but is not advised in production. In practice, the index will grow very large and will get slower to search. Deleting records based on some sort of retention period is also extremely slow.
- MonthlyIndexSharding, DailyIndexSharding, HourlyIndexSharding,
EveryMinuteIndexSharding will generate indexes based on the time of
the log. Index name is treated as a prefix. So if your index name
is
foo
and DailySharding is used, logs will be stored infoo-2016-2-25
,foo-2016-2-26
and so on. Index templating will be used to set up mappings automatically. Deletes based on date are very fast and queries can be restricted to date ranges for better performance. Queries against all dates should usefoo-*
as an index name. Note that index aliasing's glob feature is not suitable for these date ranges as it matches index names as they are declared, so new dates will be excluded. DailyIndexSharding is a reasonable choice. Changing index sharding strategies is not advisable. - CustomSharding: supply your own function that decomposes an item
into its index name heirarchy which will be appended to the index
name. So for instance if your function return ["arbitrary",
"prefix"], the index will be
foo-arbitrary-prefix
and the index template will be set to matchfoo-*
. In general, you want to use segments of increasing granularity (like year, month, day for dates). This makes it easier to address groups of indexes (e.g.foo-2016-*
).
NoIndexSharding | |
MonthlyIndexSharding | |
WeeklyIndexSharding | A special case of daily which shards to sunday |
DailyIndexSharding | |
HourlyIndexSharding | |
EveryMinuteIndexSharding | |
CustomIndexSharding (forall a. Item a -> [IndexNameSegment]) |
newtype IndexNameSegment Source
IndexNameSegment | |
|
defaultEsScribeCfg :: EsScribeCfg Source
Reasonable defaults for a config:
- defaultManagerSettings
- exponential backoff with 25ms base delay up to 5 retries
- Queue size of 1000
- Pool size of 2
- Annotate types set to False
- DailyIndexSharding
Utilities
roundToSunday :: Day -> Day Source
If the given day is sunday, returns the input, otherwise returns the previous sunday