HTTP downloader tailored for web-crawler needs.

• Handles all possible http-conduit exceptions and returns human readable error messages.
• Handles some web server bugs (returning deflate data instead of gzip, invalid gzip encoding).
• Uses OpenSSL instead of tls package (since tls doesn't handle all sites).
• Ignores invalid SSL sertificates.
• Receives data in 32k chunks internally to reduce memory fragmentation on many parallel downloads.
• Download timeout.
• Total download size limit.
• Returns HTTP headers for subsequent redownloads and handles Not modified results.
• Can be used with external DNS resolver (hsdns-cache for example).
• Keep-alive connections pool (thanks to http-conduit).

Typical workflow in crawler:

 withDnsCache $  c -> withDownloader $  d -> do
 ... -- got URL from queue
 ra <- resolveA c $ hostNameFromUrl url
 case ra of
     Left err -> ... -- uh oh, bad host
     Right ha -> do
         ... -- crawler politeness stuff (rate limits, queues)
         dr <- download d url (Just ha) opts
         case dr of
             DROK dat redownloadOptions ->
                 ... -- analyze data, save redownloadOpts for next download
             DRRedirect .. -> ...
             DRNotModified -> ...
             DRError e -> ...
 

It's highly recommended to use http://hackage.haskell.org/package/concurrent-dns-cache (preferably with single resolver pointing to locally running BIND) for DNS resolution since getAddrInfo used in http-conduit can be buggy and ineffective when it needs to resolve many hosts per second for a long time.