| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
Spark.Core.Internal.Pruning
Description
Methods to prune the computation graph.
- data NodeCacheStatus
- data NodeCacheInfo = NodeCacheInfo {}
- type NodeCache = HashMap NodeId NodeCacheInfo
- pruneGraph :: Show v => NodeCache -> (v -> NodeId) -> (v -> NodeCacheInfo -> v) -> Graph v StructureEdge -> Graph v StructureEdge
- pruneGraphDefault :: NodeCache -> Graph UntypedNode StructureEdge -> Graph UntypedNode StructureEdge
- emptyNodeCache :: NodeCache
Documentation
data NodeCacheStatus Source #
The status of a node being computed.
On purpose, it does not store data. This is meant to be only the control plane of the compuations.
Constructors
| NodeCacheRunning | |
| NodeCacheError | |
| NodeCacheSuccess |
Instances
data NodeCacheInfo Source #
This structure describes the last time a node was observed by the controller, and the state it was in.
This information is used to do smart computation pruning, by assuming that the observables are kept by the Spark processes.
Constructors
| NodeCacheInfo | |
Fields
| |
Instances
pruneGraph :: Show v => NodeCache -> (v -> NodeId) -> (v -> NodeCacheInfo -> v) -> Graph v StructureEdge -> Graph v StructureEdge Source #
It assumes a compute graph, NOT a dependency dag.