pooled-io: Run jobs on a limited number of threads and support data dependencies

The motivation for this package was to run computations on multiple cores that need to write intermediate results to disk. The functions restrict the number of simultaneously running jobs to a user given number or to the number of capabilities the Haskell program was started with, i.e. the number after the RTS option -N.

There are some flavors of this functionality:

• Control.Concurrent.PooledIO.Independent: run independent actions without results in parallel

• Control.Concurrent.PooledIO.Final: run independent actions with a final result in parallel

• Control.Concurrent.PooledIO.InOrder: run jobs in parallel with data dependencies like make -j n

Additionally there is the module Control.Concurrent.PooledIO.Sequence that helps to serialize I/O actions from multiple threads. It is certainly most useful in connection with Control.Concurrent.PooledIO.Independent.

Related packages:

• lazyio: interleave IO actions in a single thread

• async: start threads and wait for their results, forward exceptions, but do not throttle concurrency with respect to number of available cores

• parallel-tasks:

• parallel-io:

• threadPool:

• threads-pool:

• cio:

• Control-Engine:

• taskpool, async-pool:

• Control
  • Concurrent
    • PooledIO
      • Control.Concurrent.PooledIO.Final
      • Control.Concurrent.PooledIO.InOrder
      • Control.Concurrent.PooledIO.Independent
      • Control.Concurrent.PooledIO.Sequence