conduit-throttle: Throttle Conduit Producers

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

This packages is based on the throttle-io-stream package and provides functionality for throttling Conduit producers according to a provided configuration.


[Skip to Readme]

Properties

Versions 0.2.0.0, 0.2.0.1, 0.3.0.0, 0.3.0.0, 0.3.1.0
Change log None available
Dependencies async, base (>=4.7 && <5), conduit, conduit-combinators, conduit-extra, monad-control, resourcet, stm, stm-chans, throttle-io-stream, unliftio, unliftio-core [details]
License BSD-3-Clause
Copyright (c) 2017 Moritz Schulte
Author Moritz Schulte
Maintainer mtesseract@silverratio.net
Category Data
Home page https://github.com/mtesseract/conduit-throttle#readme
Bug tracker https://github.com/mtesseract/conduit-throttle/issues
Source repo head: git clone https://github.com/mtesseract/conduit-throttle
Uploaded by mtesseract at 2017-11-01T12:53:59Z

Modules

[Index]

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees


Readme for conduit-throttle-0.3.0.0

[back to package description]

conduit-throttle Hackage version Stackage version Build Status

About

This is conduit-throttle, a compact Haskell package providing configurable throttling support for Conduit. It is built on top of the packages throttle-io-stream and unliftio.

The API (Data.Conduit.Throttle) is designed for being imported qualified, which is why the exported functions and values are not prefixed by the package name or something similar.

The core function exported by this package is the following:

throttleProducer :: (MonadUnliftIO m, MonadResource m)
                 => Conf a
                 -> Producer m a
                 -> Producer m a

That is, given a configuration and a Conduit Producer, create and return a new Conduit Producer, which yields the same stream of values like the provided one but throttled according to the provided throttling configuration.

A throttling configuration is created using newConf and then configured using the functions setMeasure, setMaxThroughput, setBufferSize and others.

Primarily, the throughput is defined by the provided measure function and the configured maximum throughput. A measure function is a function which defines how "big" a Conduit element a is. In other words, it is a function of type a -> Double. For instance, it could simply calculate the length of some deserialized value in bytes or it could simply be the constant function const 1 in case it is desired to throttle the number of elements a Conduit producer produces. The maximum throughput together with the measure function then defines the maximum throughput to produce (of course it can be less, if the original producer provides data with a smaller throughput). The throughput is considered with respect to a base time interval, which can be changed using setInterval, by default it is 1000 (ms).

The underlying package throttle-io-stream uses exponentially weighted moving averages for smoothing the actual throughput changes.

Example

import qualified Data.Conduit.Throttle as ConduitThrottle

simpleProducerThrottling :: IO ()
simpleProducerThrottling = do
  let conf = ConduitThrottle.newConf
             & ConduitThrottle.setInterval 1000
             & ConduitThrottle.setMaxThroughput 1
  runResourceT . runConduit $
    ConduitThrottle.throttleProducer conf (sourceList [1..5]) .| printC