http-conduit-downloader: HTTP downloader tailored for web-crawler needs.

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

'ghc-options: -O2' is rarely needed. Check that it is giving a real benefit and not just imposing longer compile times on your users.

HTTP/HTTPS downloader built on top of http-conduit and used in https://bazqux.com crawler.

Handles all possible http-conduit exceptions and returns human readable error messages.
Handles some web server bugs (returning deflate data instead of gzip, invalid gzip encoding).
Uses OpenSSL instead of tls package (since tls doesn't handle all sites).
Ignores invalid SSL sertificates.
Receives data in 32k chunks internally to reduce memory fragmentation on many parallel downloads.
Download timeout.
Total download size limit.
Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.
Can be used with external DNS resolver (e.g. concurrent-dns-cache).

Properties

Versions	1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.0.11, 1.0.12, 1.0.13, 1.0.14, 1.0.15, 1.0.16, 1.0.17, 1.0.18, 1.0.19, 1.0.20, 1.0.21, 1.0.22, 1.0.23, 1.0.24, 1.0.25, 1.0.30, 1.0.31, 1.0.31, 1.0.32, 1.0.33, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
Change log	None available
Dependencies	base (>=4 && <5), bytestring, conduit, connection, data-default, HsOpenSSL (>=0.11.2), http-client (>=0.5.0), http-conduit (>=2.3.2), http-types, mtl, network (>=2.6), network-uri (>=2.6), resourcet, text, time (>=1.5.0), zlib [details]
License	BSD-3-Clause
Author	Vladimir Shabanov <vshabanoff@gmail.com>
Maintainer	Vladimir Shabanov <vshabanoff@gmail.com>
Category	Web
Home page	https://github.com/bazqux/http-conduit-downloader
Source repo	head: git clone https://github.com/bazqux/http-conduit-downloader
Uploaded	by VladimirShabanov at 2018-08-20T13:39:02Z

Modules

[Index] [Quick Jump]

Network
- HTTP
  - Conduit
    - Network.HTTP.Conduit.Downloader

Downloads

http-conduit-downloader-1.0.31.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

VladimirShabanov

For package maintainers and hackage trustees

edit package information