hlivy: Client library for the Apache Livy REST API.

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

The 'description' field should be longer than the 'synopsis' field. It's useful to provide an informative 'description' to allow Haskell programmers who have never heard about your package to understand the purpose of your package. The 'description' field content is typically shown by tooling (e.g. 'cabal info', Haddock, Hackage) below the 'synopsis' which serves as a headline. Please refer to <https://www.haskell.org/cabal/users-guide/developing-packages.html#package-properties> for more details.

See README.md

[Skip to Readme]

Properties

Versions	1.0.0, 1.0.0, 1.0.1
Change log	CHANGELOG.md
Dependencies	aeson (>=1.4.2 && <1.5), base (>=4.5 && <5), bytestring (>=0.10.8.2 && <0.11), exceptions (>=0.10 && <0.11), http-client (>=0.5.14 && <0.6), http-types (>=0.12.2 && <0.13), lens (>=4.17 && <5.0), mtl (>=2.2.2 && <2.3), resourcet (>=1.2.2 && <1.3), text (>=1.2.3.1 && <1.3), transformers (>=0.5.5 && <0.6), unordered-containers (>=0.2.9 && <0.3) [details]
License	MIT
Copyright	2019 Earnest Research
Author	Daniel Donohue <ddonohue@earnestresearch.com>
Maintainer	Daniel Donohue <ddonohue@earnestresearch.com>
Category	Distributed Computing
Home page	https://github.com/EarnestResearch/hlivy
Source repo	head: git clone https://github.com/EarnestResearch/hlivy.git
Uploaded	by ddonohue at 2019-02-04T18:49:01Z

Modules

[Index] [Quick Jump]

Network
- Network.Livy

Downloads

hlivy-1.0.0.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

ddonohue

For package maintainers and hackage trustees

edit package information

Readme for hlivy-1.0.0

[back to package description]

hlivy

Description

hlivy is a Haskell library that provides bindings to the Apache Livy REST API, which enables one to easily launch Spark applications -- either in an interactive or batch fashion -- via HTTP requests to the Livy server running on the master node of a Spark cluster.

Usage

Start with

import Network.Livy

which brings all functionality into scope. In particular, this exposes a monad Livy that has all the capabilites required to run Livy actions with runLivy. Generally, the format of a Livy action follows the pattern

send $ basicRequestObject requiredArg1 requiredArg2
     & requestLens1 ?~ optionalArg1
     & requestLens2 ?~ optionalArg2

This action is ran simply:

let req = basicRequestObject requiredArg1 requiredArg2
        & requestLens1 ?~ optionalArg1
        & requestLens2 ?~ optionalArg2
resp <- runLivy env $ send req

where env is a suitable environment. Concretely, if one wanted to create an interactive session, one would do something like this:

λ => import Network.Livy
λ => -- Create a default environment
λ => env <- newEnv "localhost" 8998
λ => resp <- runLivy env $ send createSession

The response body, in this case a CreateSessionResponse, should contain the the Session just created.

With this Session at hand, one can run "statements" -- snippets of Spark Scala, PySpark, SparkR, or SparkSQL -- in the given session.

λ => req = runStatement (SessionId 0) "val x = 1 + 1; println(x)" SparkSession
λ => resp <- runLivy env $ send req

This response object, in this case a RunStatementResponse, contains the information needed to check on the status of the statement or retrieve results if available.

Batch actions are organized in the Network.Livy.Client.Batch module, and are used similarly:

λ => import Control.Lens
λ => -- Application JAR in HDFS
λ => req = createBatch "/user/hadoop/my-app.jar"
λ => resp <- runLivy env (send req & cbClassName ?~ "com.company.my_app" ?~ cbExecutorCores ?~ 4)

See examples for more example use.