crawlchain-0.1.1.7: Simulation user crawl paths

Safe HaskellSafe
LanguageHaskell2010

Network.CrawlChain.CrawlDirective

Documentation

data CrawlDirective Source

Constructors

SimpleDirective (String -> [CrawlAction]) 
RelativeDirective (String -> [CrawlAction])

access content to find absolute follow-up urls

FollowUpDirective (CrawlResult -> [CrawlAction])

as simple, but found relative urls are completed

DelayDirective Int CrawlDirective

as simple, but with access to complete result

RetryDirective Int CrawlDirective

wait additional seconds before executing

AlternativeDirective CrawlDirective CrawlDirective

if given directive yields no results use add. retries

RestartChainDirective (CrawlAction, CrawlDirective)

fallback to second argument if first yields no results

GuardDirective (CrawlAction -> Bool)

the possibility to start a new chain (when using alternative)

DirectiveSequence [CrawlDirective]

not crawling anything, just a blacklisting option