pgqueuer-hs: High-performance PostgreSQL background job queue with cron and workflows

[ concurrency, database, distributed-computing, library, mit ] [ Propose Tags ] [ Report a vulnerability ]

A high-throughput, type-safe PostgreSQL-backed job queue for Haskell applications. Inspired by pgqueuer in Python, pgqueuer-hs leverages PostgreSQL's LISTEN / NOTIFY and `FOR UPDATE SKIP LOCKED` for reliable, non-contending, at-least-once distributed job processing.

Key features: * Zero-polling real-time dispatch via PostgreSQL LISTEN / NOTIFY * Non-contending job claims using `FOR UPDATE SKIP LOCKED` * Dual backend support: SimpleDb (postgresql-simple) and HasqlDb (binary protocol via Hasql) * Declarative Workflow DAG orchestration (JobTree and <~~ operator) * Native recurring cron job scheduling * Pluggable retry backoff policies (constant, linear, exponential) with full and equal jitter * Write batching and rate limiting via TimedOverflowBuffer to protect PostgreSQL WAL * Full end-to-end distributed tracing support

For getting started and runnable examples, please see the README on GitHub at https://github.com/tusharad/pgqueuer-hs#readme.


[Skip to Readme]

Flags

Manual Flags

NameDescriptionDefault
otel

Enable OpenTelemetry tracing integration

Disabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

  • No Candidates
Versions [RSS] 0.0.1.0, 0.1.0.0
Change log CHANGELOG.md
Dependencies aeson (>=2.0 && <3), async (>=2.2 && <3), base (>=4.7 && <5), bytestring (>=0.10.4.0 && <0.13), containers (>=0.6 && <0.8), contravariant (>=1.5 && <2), cron (>=0.7 && <1), hasql (>=1.6 && <1.10), hasql-pool (>=1.2 && <1.4), mtl (>=2.2 && <3), postgresql-simple (>=0.7 && <0.9), random (>=1.2 && <1.3), stm (>=2.5 && <3), text (>=1.2 && <3), time (>=1.9 && <2), unliftio (>=0.2 && <0.3), unordered-containers (>=0.2 && <0.3), uuid (>=1.3 && <2), vector (>=0.12 && <1) [details]
Tested with ghc ==9.8.4, ghc ==9.10.3
License MIT
Copyright 2026 Tushar Adhatrao
Author Tushar Adhatrao
Maintainer tusharadhatrao@gmail.com
Uploaded by tusharad at 2026-11-14T06:59:46Z
Category Database, Concurrency, Distributed Computing
Home page https://github.com/tusharad/pgqueuer-hs#readme
Bug tracker https://github.com/tusharad/pgqueuer-hs/issues
Source repo head: git clone https://github.com/tusharad/pgqueuer-hs
Distributions
Downloads 22 total (10 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs uploaded by user
Build status unknown [no reports yet]

Readme for pgqueuer-hs-0.1.0.0

[back to package description]

pgqueuer-hs

Hackage GHC License: MIT Haskell CI

A high-performance, transactional PostgreSQL-backed background job queue for Haskell, fully compatible with pgqueuer.


Overview

Your PostgreSQL database is already a distributed job queue.

pgqueuer-hs turns PostgreSQL into a fast, resilient, horizontally scalable background job processor. Jobs reside directly inside your PostgreSQL database with full ACID guarantees.

  • No separate message broker: Eliminate the operational overhead of running RabbitMQ, Redis, or Kafka for background tasks.
  • Transactional enqueuing: Enqueue jobs in the exact same database transaction as your business domain data. If your transaction rolls back, the job is never scheduled.
  • Zero-allocation binary protocol: Support for both postgresql-simple and compiled hasql prepared statements with connection pooling.
  • Cross-language compatible: Process jobs enqueued by Python pgqueuer services in Haskell, or vice versa.

Architecture

flowchart TD
    subgraph Producers ["Producers / Applications"]
        P1["App Transaction (PostgreSQL)"]
        P2["Cron Scheduler (PGQueuer)"]
        P3["DAG Workflow Root"]
    end

    subgraph Postgres ["PostgreSQL 14+"]
        Q["pgqueuer (Active Ledger)\nFOR UPDATE SKIP LOCKED"]
        L["pgqueuer_log (Unlogged Table)\nAudit & Telemetry"]
        S["pgqueuer_schedules\nDistributed Cron"]
        T["Triggers & LISTEN/NOTIFY\nInstant Dispatch"]
    end

    subgraph Workers ["Horizontal Worker Cluster"]
        W1["Worker Instance 1"]
        W2["Worker Instance 2"]
        W3["Worker Instance N"]
    end

    subgraph Buffers ["STM Decoupled Sinks"]
        B1["TimedOverflowBuffer (Logs)"]
        B2["TimedOverflowBuffer (Heartbeats)"]
    end

    P1 -->|INSERT INTO pgqueuer| Q
    P2 -->|Dispatch Due Jobs| Q
    P3 -->|insertJobTree| Q

    Q -.->|Notify Mutation| T
    T -.->|Instant Wakeup| Workers

    Workers -->|Batch Claim| Q
    Workers -->|Push Logs| B1
    Workers -->|Push Keepalives| B2

    B1 -->|Bulk Flush| L
    B2 -->|HOT Update| Q

Key Features

  • Lock-Free Concurrency: Leverages FOR UPDATE SKIP LOCKED and partial indexes to eliminate row lock contention across horizontal worker pools.
  • Dual Backend Architecture (MonadPGQueuer):
    • SimpleDb: Reference driver built on postgresql-simple for lightweight setups.
    • HasqlDb: High-performance binary driver using server-side prepared statements and hasql-pool.
  • Instant Event Dispatch: Emits PostgreSQL LISTEN / NOTIFY signals when jobs arrive, waking workers immediately without wasteful high-frequency polling.
  • STM Timed Overflow Buffering: Worker threads push status transitions and heartbeats to bounded STM queues (TimedOverflowBuffer), flushed in bulk to eliminate WAL write amplification.
  • Resilient Backoff & Jitter: Configurable mathematical backoff strategies (Exponential, Linear, Constant) with anti-thundering-herd jitter (FullJitter, EqualJitter, NoJitter).
  • DAG Workflow Trees: Build complex job dependency pipelines (JobTree, <~~). Parent jobs remain held in PostgreSQL until all child tasks finish successfully.
  • Distributed Cron Scheduling: Native 5-field cron recurring jobs (CronExpression) stored and coordinated directly in PostgreSQL.
  • Distributed Tracing (OpenTelemetry): Built-in W3C trace context injection and span extraction in job headers (-f otel).

Installation

Add pgqueuer-hs to your package.yaml:

dependencies:
  - pgqueuer-hs >= 0.1.0.0
  - postgresql-simple
  - aeson
  - text
  - time
  - uuid

Or in your .cabal file:

build-depends:
    base >= 4.7 && < 5
  , pgqueuer-hs >= 0.1.0.0

Quick Start (15 Lines)

{-# LANGUAGE OverloadedStrings #-}

import Data.Aeson (encode)
import Data.UUID.V4 (nextRandom)
import PGQueuer

main :: IO ()
main = do
  let connStr = "postgresql://queue_user:queue_pass@localhost:5432/queue_db"
  queueMgrId <- nextRandom

  withQueueManager connStr defaultDBSettings queueMgrId $ \qm -> do
    -- 1. Setup schema automatically
    setupSchema qm

    -- 2. Register entrypoint handler
    let ep = Entrypoint "SendWelcomeEmail"
    qm1 <- registerEntrypoint qm ep $ \job -> do
      putStrLn $ "Processing job #" ++ show (let JobId jid = jobId job in jid)

    -- 3. Enqueue a job
    _ <- enqueue qm1 ep (Just $ encode ("user@example.com" :: String)) 0 Nothing Nothing Nothing

    -- 4. Start worker loop (blocks and processes jobs)
    putStrLn "Worker listening for incoming jobs..."
    let execParams = [EntrypointExecutionParameter ep 5 3 (Exponential 2 60) FullJitter]
    workerLoop qm1 defaultBufferConfig execParams

Backend Comparison

pgqueuer-hs provides a unified MonadPGQueuer typeclass that abstracts low-level database operations:

Feature SimpleDb (postgresql-simple) HasqlDb (hasql)
Protocol Text/Parametrized Compiled Binary Protocol
Connection Pooling Single Connection or external pool Native hasql-pool (built-in)
Query Planning Client-side query preparation Server-side prepared statements
Best For Simple CLI scripts, prototypes, testing High-throughput production microservices

Running with Hasql Connection Pool

import Database.PostgreSQL.Simple (connectPostgreSQL)
import PGQueuer.Backend.Hasql
import PGQueuer.Core.Monad
import PGQueuer.Schema (install)
import PGQueuer.Settings (defaultDBSettings)
import PGQueuer.Types

main :: IO ()
main = do
  let connStr = "postgresql://queue_user:queue_pass@localhost:5432/queue_db"
  env <- mkHasqlDbEnv connStr defaultDBSettings

  -- Execute transactional job operations
  runHasqlDb env $ withTransaction $ do
    _ <- enqueue (Entrypoint "ImageResize") Nothing 0 Nothing Nothing Nothing Nothing Nothing Queued
    pure ()

  releaseHasqlDbEnv env

Core Capabilities

1. Resilient Retries with Jitter

Prevent cascading failures and thundering herd stampedes by applying backoff and jitter algorithms:

let params =
      [ EntrypointExecutionParameter
          { paramEntrypoint = Entrypoint "ExternalPaymentAPI"
          , paramConcurrencyLimit = 5       -- At most 5 concurrent jobs
          , paramMaxAttempts = 5            -- Retry up to 5 times
          , paramBackoffStrategy = Exponential 2 60 -- Base 2s, Cap 60s
          , paramJitter = EqualJitter       -- Half deterministic, half random
          }
      ]

2. DAG Workflows & Dependency Trees

Define workflows where parent tasks wait for child tasks to complete:

-- Define child task nodes
let childA = JobNode (Entrypoint "ExtractData") (Just "chunk-1") 0 Nothing Nothing Nothing Nothing <~~ []
    childB = JobNode (Entrypoint "ExtractData") (Just "chunk-2") 0 Nothing Nothing Nothing Nothing <~~ []

-- Parent MasterReport task depends on childA and childB
let workflow = JobNode (Entrypoint "MasterReport") Nothing 0 Nothing Nothing Nothing Nothing <~~ [childA, childB]

-- Atomically inserts tree: children are Queued, parent is Held until children complete
insertJobTree workflow

3. Distributed Cron Scheduling

Coordinate periodic jobs across your entire cluster without external daemons:

-- Register a schedule running every 15 minutes
registerSchedule qm (CronExpression "*/15 * * * *") (Entrypoint "SyncCatalog")

Examples

Fully working examples are located in the example directory:

  • 01-hello-mailer: Basic queue management, JSON payload handling, worker loop.
  • 02-resilient-scraper: Flaky API handling with JobRetryableException, exponential backoff, equal jitter, and admin retry watchdog.
  • 03-map-reduce-etl: Hierarchical DAG workflows (JobTree) with 500 child tasks and parent state aggregation.
  • 04-chaos-ecommerce: High-throughput stress test with 5,000+ jobs/sec and a live queue status dashboard.
  • 05-hasql-worker: Zero-allocation binary protocol backend with connection pooling and FOR UPDATE SKIP LOCKED.
  • 06-cron-pipeline: Distributed cron schedule management and periodic worker dispatch.

To run the examples:

# 1. Start local PostgreSQL
make db-up

# 2. Run any example
cd example
stack run hello-mailer
stack run hasql-worker
stack run cron-pipeline

Development & Makefile

make help          # List all available targets
make db-up         # Start local PostgreSQL container (Podman or Docker)
make test          # Run test suite against PostgreSQL
make bench         # Compile and execute benchmarks
make lint          # Run HLint static analysis
make format        # Format all Haskell files with Fourmolu
make format-check  # Verify formatting compliance (CI check)
make check         # Run cabal check for Hackage packaging
make docs          # Build Haddock documentation
make clean         # Clean build artifacts
make db-down       # Stop and remove local database container

License

This project is licensed under the MIT License - see the LICENSE file for details.


See Also