Hackage :: [Package]

hlivy: Client library for the Apache Livy REST API

[ distributed-computing, library, mit ] [ Propose Tags ]
Versions [RSS] 1.0.0, 1.0.1
Change log CHANGELOG.md
Dependencies aeson (>=1.1.2 && <1.5), base (>=4.7 && <5), bytestring (>=0.9 && <0.11), exceptions (>=0.6 && <0.11), http-client (>=0.4 && <0.6), http-types (>=0.9.1 && <0.13), lens (>=4.4 && <5.0), mtl (>=2.2.1 && <2.3), resourcet (>=1.1 && <1.3), text (>=1.1 && <1.3), transformers (>=0.5.2 && <0.6), unordered-containers (>=0.2.7 && <0.3) [details]
License MIT
Copyright 2019 Earnest Research
Author Daniel Donohue <ddonohue@earnestresearch.com>
Maintainer Daniel Donohue <ddonohue@earnestresearch.com>
Category Distributed Computing
Home page https://github.com/EarnestResearch/hlivy
Source repo head: git clone https://github.com/EarnestResearch/hlivy.git
Uploaded by ddonohue at 2019-02-08T23:43:27Z
Distributions
Downloads 969 total (5 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2019-02-09 [all 1 reports]

Readme for hlivy-1.0.1

[back to package description]

hlivy

Build Status Hackage Version

Description

hlivy is a Haskell library that provides bindings to the Apache Livy REST API, which enables one to easily launch Spark applications -- either in an interactive or batch fashion -- via HTTP requests to the Livy server running on the master node of a Spark cluster.

Usage

Start with

import Network.Livy

which brings all functionality into scope. In particular, this exposes a monad Livy that has all the capabilites required to run Livy actions with runLivy. Generally, the format of a Livy action follows the pattern

send $ basicRequestObject requiredArg1 requiredArg2
     & requestLens1 ?~ optionalArg1
     & requestLens2 ?~ optionalArg2

This action is ran simply:

let req = basicRequestObject requiredArg1 requiredArg2
        & requestLens1 ?~ optionalArg1
        & requestLens2 ?~ optionalArg2
resp <- runLivy env $ send req

where env is a suitable environment. Concretely, if one wanted to create an interactive session, one would do something like this:

λ => import Network.Livy
λ => -- Create a default environment
λ => env <- newEnv "localhost" 8998
λ => resp <- runLivy env $ send createSession

The response body, in this case a CreateSessionResponse, should contain the the Session just created.

With this Session at hand, one can run "statements" -- snippets of Spark Scala, PySpark, SparkR, or SparkSQL -- in the given session.

λ => req = runStatement (SessionId 0) "val x = 1 + 1; println(x)" SparkSession
λ => resp <- runLivy env $ send req

This response object, in this case a RunStatementResponse, contains the information needed to check on the status of the statement or retrieve results if available.

Batch actions are organized in the Network.Livy.Client.Batch module, and are used similarly:

λ => import Control.Lens
λ => -- Application JAR in HDFS
λ => req = createBatch "/user/hadoop/my-app.jar"
λ => resp <- runLivy env (send req & cbClassName ?~ "com.company.my_app" ?~ cbExecutorCores ?~ 4)

See examples for more example use.