% A wreq tutorial % Learn how to write web clients. We start easy, then ramp up the power. # Installation To use the `wreq` package, simply use `cabal`, the standard Haskell package management command. ~~~~ cabal update cabal install -j --disable-tests wreq ~~~~ Depending on how many prerequisites you already have installed, and what your Cabal configuration looks like, the build may take a few minutes: a few seconds for `wreq`, and the rest for its dependencies. # Interactive usage We'll run our examples interactively via the `ghci` shell. ~~~~ $ ghci ~~~~ To start using `wreq`, we import the [`Network.Wreq`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html) module. ~~~~ {.haskell} ghci> import Network.Wreq ghci> r <- get "http://httpbin.org/get" ghci> :type r r :: Response ByteString ~~~~ The variable `r` above is the [`Response`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:Response) from the server. ## Working with string-like types Complex Haskell libraries and applications have to deal fluently with Haskell's three main string types: `String` ("legacy"), `Text`, and `ByteString` (mostly used for binary data, sometimes ASCII). To write string literals without having to always provide a conversion function, we use the `OverloadedStrings` language extension. Throughout the rest of this tutorial, we'll assume that you have enabled `OverloadedStrings` in `ghci`: ~~~~ {.haskell} ghci> :set -XOverloadedStrings ~~~~ If you're using `wreq` from a Haskell source file, put a pragma at the top of your file: ~~~~ {.haskell} {-# LANGUAGE OverloadedStrings #-} ~~~~ # A quick lens backgrounder The `wreq` package makes heavy use of Edward Kmett's [`lens`](https://lens.github.io/) package to provide a clean, consistent API. ~~~~ {.haskell} ghci> import Control.Lens ~~~~ While `lens` has a vast surface area, the portion that you must understand in order to productively use `wreq` is tiny. A lens provides a way to focus on a portion of a Haskell value. For example, the `Response` type has a [`responseStatus`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:responseStatus) lens, which focuses on the status information returned by the server. ~~~~ {.haskell} ghci> r ^. responseStatus Status {statusCode = 200, statusMessage = "OK"} ~~~~ The [`^.`](http://hackage.haskell.org/package/lens/docs/Control-Lens-Getter.html#v:-94-.) operator takes a value as its first argument, a lens as its second, and returns the portion of the value focused on by the lens. We compose lenses using function composition, which allows us to easily focus on part of a deeply nested structure. ~~~~ {.haskell} ghci> r ^. responseStatus . statusCode 200 ~~~~ We'll have more to say about lenses as this tutorial proceeds. # Changing default behaviours While [`get`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:get) is convenient and easy to use, there's a lot more power available to us. For example, if we want to add parameters to the query string of a URL, we will use the [`getWith`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:getWith) function. The `*With` family of functions all accept an [`Options`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:Options) parameter that allow changes from the library's default behaviours. ~~~~ {.haskell} ghci> import Data.Aeson.Lens (_String, key) ghci> let opts = defaults & param "foo" .~ ["bar", "quux"] ghci> r <- getWith opts "http://httpbin.org/get" ghci> r ^. responseBody . key "url" . _String "http://httpbin.org/get?foo=bar&foo=quux" ~~~~ (We'll talk more about `key` and `_String` below.) The default parameters for all queries is represented by the variable [`defaults`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:defaults). (In fact, `get` is defined simply as `getWith defaults`.) Here's where we get to learn a little more about lenses. In addition to *getting* a value from a nested structure, we can also *set* (edit) a value within a nested structure, which makes an identical copy of the structure except for the portion we want to modify. The `&` operator is just function application with its operands reversed, so the function is on the right and its parameter is on the left. ~~~~ {.haskell} parameter & functionToApply ~~~~ The [`.~`](http://hackage.haskell.org/package/lens/docs/Control-Lens-Setter.html#v:.-126-) operator turns a lens into a setter function, with the lens on the left and the new value on the right. ~~~~ {.haskell} param "foo" .~ ["bar", "quux"] ~~~~ The [`param`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:param) lens focuses on the values associated with the given key in the query string. ~~~~ {.haskell} param :: Text -> Lens' Options [Text] ~~~~ The reason we allow for a list of values instead of just a single value is simply that this is completely legitimate. For instance, in our example above we generate the query string `foo=bar&foo=quux`. If you use non-ASCII characters in a `param` key or value, they will be encoded as UTF-8 before being URL-encoded, so that they can be safely transmitted over the wire. # Accessing the body of a response The [`responseBody`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:responseBody) lens gives us access to the body of a response. ~~~~ {.haskell} ghci> r <- get "http://httpbin.org/get" ghci> r ^. responseBody "{\n \"headers\": {\n \"Accept-Encoding\": \"gzip"{-...-} ~~~~ The response body is a raw lazy [`ByteString`](http://hackage.haskell.org/package/bytestring/docs/Data-ByteString-Lazy.html#t:ByteString). ## JSON responses We can use the [`asJSON`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:asJSON) function to convert a response body to a Haskell value that implements the [`FromJSON`](http://hackage.haskell.org/package/aeson/docs/Data-Aeson-Types.html#t:FromJSON) class. ~~~~ {.haskell} ghci> import Data.Map as Map ghci> import Data.Aeson (Value) ghci> type Resp = Response (Map String Value) ghci> r <- asJSON =<< get "http://httpbin.org/get" :: IO Resp ghci> Map.size (r ^. responseBody) 4 ~~~~
In this example, we have to tell `ghci` exactly what target type we are expecting. In a real Haskell program, the correct return type will usually be inferred automatically, making an explicit type signature unnecessary in most cases.
If the response is not `application/json`, or we try to convert to an incompatible Haskell type, a [`JSONError`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:JSONError) exception will be thrown. ~~~~ {.haskell} ghci> type Resp = Response [Int] ghci> r <- asJSON =<< get "http://httpbin.org/get" :: IO Resp *** Exception: JSONError "when expecting a [a], encountered Object instead" ~~~~ ## Convenient JSON traversal The `lens` package provides some extremely useful functions for traversing JSON structures without having to either build a corresponding Haskell type or traverse a `Value` by hand. The first of these is [`key`](http://hackage.haskell.org/package/lens/docs/Data-Aeson-Lens.html#v:key), which traverses to the named key in a JSON object. ~~~~ {.haskell} ghci> import Data.Aeson.Lens (key) ghci> r <- get "http://httpbin.org/get" ghci> r ^? responseBody . key "url" Just (String "http://httpbin.org/get") ~~~~
Notice our use of the [`^?`](http://hackage.haskell.org/package/lens-4.1.2/docs/Control-Lens-Fold.html#v:-94--63-) operator here. This is like `^.`, but it allows for the possibility that an access might fail---and of course there may not be a key named `"url"` in our object.
That said, our result above has the type `Maybe Value`, so it's quite annoying to work with. This is where the `_String` lens comes in. ~~~~ {.haskell} ghci> import Data.Aeson.Lens (_String, key) ghci> r <- get "http://httpbin.org/get" ghci> r ^. responseBody . key "url" . _String "http://httpbin.org/get" ~~~~ If the key exists, and is a `Value` with a `String` constructor, `_String` gives us back a regular `Text` value with all the wrappers removed; otherwise it gives an empty value. Notice what happens as we switch between `^?` and `^.` in these examples. ~~~~ {.haskell} ghci> r ^. responseBody . key "fnord" . _String "" ghci> r ^? responseBody . key "fnord" . _String Nothing ghci> r ^? responseBody . key "url" . _String Just "http://httpbin.org/get" ~~~~ # Working with headers To add headers to a request, we use the [`header`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:header) lens. ~~~~ {.haskell} ghci> let opts = defaults & header "Accept" .~ ["application/json"] ghci> getWith opts "http://httpbin.org/get" ~~~~ As with the [`param`](#param) lens, if we provide more than one value to go with a single key, this will expand to several headers. ~~~~ {.haskell} header :: HeaderName -> Lens' Options [ByteString] ~~~~ When we want to inspect the headers of a response, we use the [`responseHeader`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:responseHeader) lens. ~~~~ {.haskell} ghci> r <- get "http://httpbin.org/get" ghci> r ^. responseHeader "content-type" "application/json" ~~~~
Header names are case insensitive.
If a header is not present in a response, then `^.` will give an empty string, while `^?` will give `Nothing`. ~~~~ {.haskell} ghci> r ^. responseHeader "X-Nonesuch" "" ghci> r ^? responseHeader "X-Nonesuch" Nothing ~~~~ # Uploading data via POST We use the [`post`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:post) and [`postWith`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:postWith) functions to issue POST requests. ~~~~ {.haskell} ghci> r <- post "http://httpbin.org/post" ["num" := 3, "str" := "wat"] ghci> r ^? responseBody . key "form" Just (Object fromList [("num",String "3"),("str",String "wat")]) ~~~~ The [httpbin.org](http://httpbin.org/) server conveniently echoes our request headers back at us, so we can see what kind of body we POSTed. ~~~~ {.haskell} ghci> r ^. responseBody . key "headers" . key "Content-Type" . _String "application/x-www-form-urlencoded" ~~~~ The [`:=`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v::-61-) operator is the constructor for the [`FormParam`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:FormParam) type, which `wreq` uses as a key/value pair to generate an `application/x-www-form-urlencoded` form body to upload. A class named [`FormValue`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:FormValue) determines how the operand on the right-hand side of `:=` is encoded, with sensible default behaviours for strings and numbers. The slightly more modern way to upload POST data is via a `multipart/form-data` payload, for which `wreq` provides the [`Part`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:Part) type. ~~~~ {.haskell} ghci> r <- post "http://httpbin.org/post" [partText "button" "o hai"] ghci> r ^. responseBody . key "headers" . key "Content-Type" . _String "multipart/form-data; boundary=----WebKitFormBoundaryJsEZfuj89uj" ~~~~ The first argument to these `part*` functions is the label of the `` element in the form being uploaded. Let's inspect httpbin.org's response to see what we uploaded. When we think there could be more than one value associated with a lens, we use the [`^..`](http://hackage.haskell.org/package/lens-4.1.2/docs/Control-Lens-Fold.html#v:-94-..) operator, which returns a list. ~~~~ {.haskell} ghci> r ^.. responseBody . key "form" [Object fromList [("button",String "o hai")]] ~~~~ ## Uploading file contents To upload a file as part of a `multipart/form-data` POST, we use [`partFile`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:partFile), or if the file is large enough that we want to stream its contents, [`partFileSource`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:partFileSource). ~~~~ {.haskell} ghci> import Data.Aeson.Lens (members) ghci> r <- post "http://httpbin.org/post" (partFile "file" "hello.hs") ghci> r ^.. responseBody . key "files" . members . _String ["main = putStrLn \"hello\"\n"] ~~~~ Both `partFile` and `partFileSource` will set the filename of a part to whatever name they are given, and guess its content-type based on the file name extension. Here's an example of how we can upload a file without revealing its name. ~~~~ {.haskell} ghci> partFile "label" "foo.hs" & partFileName .~ Nothing Part "label" Nothing (Just "text/plain") ~~~~ # Cookies To see how easily we can work with cookies, let's ask the ever-valuable httpbin.org to set a cookie in a response. ~~~~ {.haskell} ghci> r <- get "http://httpbin.org/cookies/set?foo=bar" ghci> r ^. responseCookie "foo" . cookieValue "bar" ~~~~ To make cookies even easier to deal with, we'll want to [use the `Session` API](#session), but we'll come back to that later. # Authentication The `wreq` library supports both basic authentication and OAuth2 bearer authentication.
**Note:** the security of these mechanisms is _absolutely dependent on your use of TLS_, as the credentials can easily be stolen and reused if transmitted unencrypted.
If we try to access a service that requires authentication, `wreq` will throw a [`HttpException`](http://hackage.haskell.org/package/http-client/docs/Network-HTTP-Client.html#t:HttpException). ~~~~ {.haskell} ghci> r <- get "http://httpbin.org/basic-auth/user/pass" *** Exception: HttpExceptionRequest Request { ... } (StatusCodeException (Response { responseStatus = Status {statusCode = 401, {-...-} } , {- ... -} }), "..." ) ~~~~ If we then supply a username and password, our request will succeed. (Notice that we follow our own advice: we switch to `https` for our retry.) ~~~~ {.haskell} ghci> let opts = defaults & auth ?~ basicAuth "user" "pass" ghci> r <- getWith opts "https://httpbin.org/basic-auth/user/pass" ghci> r ^. responseBody "{\n \"authenticated\": true,\n \"user\": \"user\"\n}" ~~~~
We use the [`?~`](http://hackage.haskell.org/package/lens/docs/Control-Lens-Setter.html#v:-63--126-) operator to turn an [`Auth`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#t:Auth) into a `Maybe Auth` here, to make the type of value on the right hand side compatible with the [`auth`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:auth) lens.
For OAuth2 bearer authentication, `wreq` supports two flavours: [`oauth2Bearer`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:oauth2Bearer) is the standard bearer token, while [`oauth2Token`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:oauth2Token) is GitHub's variant. These tokens are equivalent in value to a username and password. ## Amazon Web Services (AWS) To authenticate to Amazon Web Services (AWS), we use [`awsAuth`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq.html#v:awsAuth). In this example, we set the `Accept` header to request JSON, as opposed to XML output from AWS. ~~~~ {.haskell} ghci> let opts = defaults & auth ?~ awsAuth AWSv4 "key" "secret" & header "Accept" .~ ["application/json"] ghci> r <- getWith opts "https://sqs.us-east-1.amazonaws.com/?Action=ListQueues" ghci> r ^. responseBody "{\"ListQueuesResponse\":{\"ListQueuesResult\":{\"queueUrls\": ... }" ~~~~ ## Runscope support for Amazon Web Services (AWS) requests To send requests to AWS through the [Runscope Inc.](https://www.runscope.com) Traffic Inspector, convert the AWS service URL to a Runscope Bucket URL using the "URL Helper" section in the Runscope dashboard (as you would for other HTTP endpoints). Then invoke the AWS service as before. For example, if your Runscope bucket key is `7kh11example`, call AWS like so: ~~~~ {.haskell} ghci> let opts = defaults & auth ?~ awsAuth AWSv4 "key" "secret" & header "Accept" .~ ["application/json"] ghci> r <- getWith opts "https://sqs-us--east--1-amazonaws-com-7kh11example.runscope.net/?Action=ListQueues" ghci> r ^. responseBody "{\"ListQueuesResponse\":{\"ListQueuesResult\":{\"queueUrls\": ... }" ~~~~ If you enabled "Require Authentication Token" in the "Bucket Settings" of your Runscope dashboard, set the `Runscope-Bucket-Auth` header like so: ~~~~ {.haskell} ghci> let opts = defaults & auth ?~ awsAuth AWSv4 "key" "secret" & header "Accept" .~ ["application/json"] & header "Runscope-Bucket-Auth" .~ ["1example-1111-4yyyy-zzzz-xxxxxxxx"] ghci> r <- getWith opts "https://sqs-us--east--1-amazonaws-com-7kh11example.runscope.net/?Action=ListQueues" ghci> r ^. responseBody "{\"ListQueuesResponse\":{\"ListQueuesResult\":{\"queueUrls\": ... }" ~~~~ # Error handling Most of the time when an error occurs or a request fails, `wreq` will throw a `HttpException`. ~~~~ {.haskell} h> r <- get "http://httpbin.org/wibblesticks" *** Exception: HttpExceptionRequest Request { ... } (StatusCodeException (Response { responseStatus = Status {statusCode = 404, {-...-} } , {- ... -} }), "..." ) ~~~~ Here's a simple example of how we can respond to one kind of error: a `get`-like function that retries with authentication if an unauthenticated request fails. ~~~~ {.haskell} import Control.Exception as E import Control.Lens import Network.HTTP.Client (HttpException (HttpExceptionRequest), HttpExceptionContent (StatusCodeException)) import Network.Wreq getAuth url myauth = get url `E.catch` handler where handler e@(HttpExceptionRequest _ (StatusCodeException r _)) | r ^. responseStatus . statusCode == 401 = getWith authopts authurl | otherwise = throwIO e handler e = throwIO e authopts = defaults & auth ?~ myauth -- switch to TLS when we use auth authurl = "https" ++ dropWhile (/=':') url ~~~~ (A "real world" version would remember which URLs required authentication during a session, to avoid the need for an unauthenticated failure followed by an authenticated success if we visit the same endpoint repeatedly.) # Handling multiple HTTP requests For non-trivial applications, we'll always want to use a [`Session`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq-Session.html#t:Session) to efficiently and correctly handle multiple requests. The `Session` API provides two important features: * When we issue multiple HTTP requests to the same server, a `Session` will reuse TCP and TLS connections for us. (The simpler API we've discussed so far does not do this.) This greatly improves efficiency. * A `Session` transparently manages HTTP cookies. (We can manage them by hand, but it's awkward and verbose, so we won't cover it in this tutorial.) Here's a complete example. ~~~~ {.haskell} {-# LANGUAGE OverloadedStrings #-} import Control.Lens import Network.Wreq import qualified Network.Wreq.Session as S main :: IO () main = do sess <- S.newSession -- First request: tell the server to set a cookie S.get sess "http://httpbin.org/cookies/set?name=hi" -- Second request: the cookie should still be set afterwards. r <- S.post sess "http://httpbin.org/post" ["a" := (3 :: Int)] print $ r ^. responseCookie "name" . cookieValue ~~~~ The key differences from the basic API are as follows. * We import the [`Network.Wreq.Session`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq-Session.html) module qualified, and we'll identify its functions by prefixing them with "`S.`". * To create a `Session`, we use `S.newSession`. * Instead of `get` and `post`, we call the `Session`-specific versions, [`S.get`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq-Session.html#v:get) and [`S.post`](http://hackage.haskell.org/package/wreq/docs/Network-Wreq-Session.html#v:post), and pass `sess` to each of them.