Safe Haskell	None
Language	Haskell2010

Spark.Core.Internal.ContextStructures

Synopsis

Documentation

The configuration of a remote spark session in Karps.

Constructors

Fields

confEndPoint :: !Text
The URL of the end point.
confPort :: !Int
The port used to configure the end point.
confPollingIntervalMillis :: !Int
(internal) the polling interval
confRequestedSessionName :: !Text
(optional) the requested name of the session. This name must obey a number of rules: - it must consist in alphanumerical and -,_: [a-zA-Z0-9-_] - if it already exists on the server, it will be reconnected to
The default value is "" (a new random context name will be chosen).
confUseNodePrunning :: !Bool
If enabled, attempts to prune the computation graph as much as possible.
This option is useful in interactive sessions when long chains of computations are extracted. This forces the execution of only the missing parts. The algorithm is experimental, so disabling it is a safe option.
Disabled by default.

Instances

Show SparkSessionConf Source #
Methods showsPrec :: Int -> SparkSessionConf -> ShowS # show :: SparkSessionConf -> String # showList :: [SparkSessionConf] -> ShowS #

A session in Spark. Encapsualates all the state needed to communicate with Spark and to perfor some simple optimizations on the code.

Constructors

SparkSession
Fields ssConf :: !SparkSessionConf ssId :: !LocalSessionId ssCommandCounter :: !Integer ssNodeCache :: !NodeCache

Instances

Show SparkSession Source #
Methods showsPrec :: Int -> SparkSession -> ShowS # show :: SparkSession -> String # showList :: [SparkSession] -> ShowS #

Represents the state of a session and accounts for the communication with the server.

internal

A graph of computations. This graph is a direct acyclic graph. Each node is associated to a global path.

A path in the Hadoop File System (HDFS).

These paths are usually not created by the user directly.

Constructors

Instances

Eq HdfsPath Source #
Methods (==) :: HdfsPath -> HdfsPath -> Bool # (/=) :: HdfsPath -> HdfsPath -> Bool #
Ord HdfsPath Source #
Methods compare :: HdfsPath -> HdfsPath -> Ordering # (<) :: HdfsPath -> HdfsPath -> Bool # (<=) :: HdfsPath -> HdfsPath -> Bool # (>) :: HdfsPath -> HdfsPath -> Bool # (>=) :: HdfsPath -> HdfsPath -> Bool # max :: HdfsPath -> HdfsPath -> HdfsPath # min :: HdfsPath -> HdfsPath -> HdfsPath #
Show HdfsPath Source #
Methods showsPrec :: Int -> HdfsPath -> ShowS # show :: HdfsPath -> String # showList :: [HdfsPath] -> ShowS #
ToJSON HdfsPath Source #
Methods toJSON :: HdfsPath -> Value # toEncoding :: HdfsPath -> Encoding #
FromJSON HdfsPath Source #
Methods parseJSON :: Value -> Parser HdfsPath #

This structure describes the last time a node was observed by the controller, and the state it was in.

This information is used to do smart computation pruning, by assuming that the observables are kept by the Spark processes.

Constructors

NodeCacheInfo
Fields nciStatus :: !NodeCacheStatus nciComputation :: !ComputationID nciPath :: !NodePath

Instances

Eq NodeCacheInfo Source #
Methods (==) :: NodeCacheInfo -> NodeCacheInfo -> Bool # (/=) :: NodeCacheInfo -> NodeCacheInfo -> Bool #
Show NodeCacheInfo Source #
Methods showsPrec :: Int -> NodeCacheInfo -> ShowS # show :: NodeCacheInfo -> String # showList :: [NodeCacheInfo] -> ShowS #

The status of a node being computed.

On purpose, it does not store data. This is meant to be only the control plane of the compuations.

Constructors

Instances

Eq NodeCacheStatus Source #
Methods (==) :: NodeCacheStatus -> NodeCacheStatus -> Bool # (/=) :: NodeCacheStatus -> NodeCacheStatus -> Bool #
Show NodeCacheStatus Source #
Methods showsPrec :: Int -> NodeCacheStatus -> ShowS # show :: NodeCacheStatus -> String # showList :: [NodeCacheStatus] -> ShowS #