| Copyright | (c) Ian Duncan 2014 |
|---|---|
| License | MIT |
| Maintainer | ian@iankduncan.com |
| Stability | unstable |
| Portability | non-portable |
| Safe Haskell | None |
| Language | Haskell2010 |
System.Mesos.Scheduler
Contents
Description
Mesos scheduler interface and scheduler driver. A scheduler is used to interact with Mesos in order run distributed computations.
- class ToScheduler a where
- registered :: a -> SchedulerDriver -> FrameworkID -> MasterInfo -> IO ()
- reRegistered :: a -> SchedulerDriver -> MasterInfo -> IO ()
- disconnected :: a -> SchedulerDriver -> IO ()
- resourceOffers :: a -> SchedulerDriver -> [Offer] -> IO ()
- offerRescinded :: a -> SchedulerDriver -> OfferID -> IO ()
- statusUpdate :: a -> SchedulerDriver -> TaskStatus -> IO ()
- frameworkMessage :: a -> SchedulerDriver -> ExecutorID -> SlaveID -> ByteString -> IO ()
- slaveLost :: a -> SchedulerDriver -> SlaveID -> IO ()
- executorLost :: a -> SchedulerDriver -> ExecutorID -> SlaveID -> Status -> IO ()
- errorMessage :: a -> SchedulerDriver -> ByteString -> IO ()
- data SchedulerDriver
- withSchedulerDriver :: ToScheduler a => a -> FrameworkInfo -> ByteString -> Maybe Credential -> (SchedulerDriver -> IO b) -> IO b
- start :: SchedulerDriver -> IO Status
- stop :: SchedulerDriver -> Bool -> IO Status
- abort :: SchedulerDriver -> IO Status
- await :: SchedulerDriver -> IO Status
- run :: SchedulerDriver -> IO Status
- requestResources :: SchedulerDriver -> [Request] -> IO Status
- launchTasks :: SchedulerDriver -> [OfferID] -> [TaskInfo] -> Filters -> IO Status
- killTask :: SchedulerDriver -> TaskID -> IO Status
- declineOffer :: SchedulerDriver -> OfferID -> Filters -> IO Status
- reviveOffers :: SchedulerDriver -> IO Status
- sendFrameworkMessage :: SchedulerDriver -> ExecutorID -> SlaveID -> ByteString -> IO Status
- reconcileTasks :: SchedulerDriver -> [TaskStatus] -> IO Status
- createDriver :: Scheduler -> FrameworkInfo -> ByteString -> Maybe Credential -> IO SchedulerDriver
- destroyDriver :: SchedulerDriver -> IO ()
- data Scheduler
- createScheduler :: ToScheduler a => a -> IO Scheduler
- destroyScheduler :: Scheduler -> IO ()
Creating a new Scheduler
class ToScheduler a where Source
Callback interface to be implemented by frameworks' schedulers. Note that only one callback will be invoked at a time, so it is not recommended that you block within a callback because it may cause a deadlock.
Minimal complete definition
Nothing
Methods
registered :: a -> SchedulerDriver -> FrameworkID -> MasterInfo -> IO () Source
Invoked when the Scheduler successfully registers with a Mesos
master. A unique ID (generated by the master) used for
distinguishing this framework from others and MasterInfo
with the ip and port of the current master are provided as arguments.
reRegistered :: a -> SchedulerDriver -> MasterInfo -> IO () Source
Invoked when the Scheduler re-registers with a newly elected Mesos master.
This is only called when the scheduler has previously been registered.
MasterInfo containing the updated information about the elected master
is provided as an argument.
disconnected :: a -> SchedulerDriver -> IO () Source
Invoked when the Scheduler becomes "disconnected" from the master
(e.g., the master fails and another is taking over).
resourceOffers :: a -> SchedulerDriver -> [Offer] -> IO () Source
Invoked when resources have been offered to this framework. A
single offer will only contain resources from a single slave.
Resources associated with an offer will not be re-offered to
_this_ framework until either (a) this framework has rejected
those resources (see launchTasks) or (b) those
resources have been rescinded (see offerRescinded).
Note that resources may be concurrently offered to more than one
framework at a time (depending on the allocator being used). In
that case, the first framework to launch tasks using those
resources will be able to use them while the other frameworks
will have those resources rescinded (or if a framework has
already launched tasks with those resources then those tasks will
fail with a Lost status and a message saying as much).
offerRescinded :: a -> SchedulerDriver -> OfferID -> IO () Source
Invoked when an offer is no longer valid (e.g., the slave was
lost or another framework used resources in the offer). If for
whatever reason an offer is never rescinded (e.g., dropped
message, failing over framework, etc.), a framwork that attempts
to launch tasks using an invalid offer will receive Lost
status updates for those tasks (see resourceOffers).
statusUpdate :: a -> SchedulerDriver -> TaskStatus -> IO () Source
Invoked when the status of a task has changed (e.g., a slave is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc). Note that returning from this callback _acknowledges_ receipt of this status update! If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered (note, however, that this is currently not true if the slave sending the status update is lost/fails during that time).
frameworkMessage :: a -> SchedulerDriver -> ExecutorID -> SlaveID -> ByteString -> IO () Source
Invoked when an executor sends a message. These messages are best effort; do not expect a framework message to be retransmitted in any reliable fashion.
slaveLost :: a -> SchedulerDriver -> SlaveID -> IO () Source
Invoked when a slave has been determined unreachable (e.g., machine failure, network partition). Most frameworks will need to reschedule any tasks launched on this slave on a new slave.
executorLost :: a -> SchedulerDriver -> ExecutorID -> SlaveID -> Status -> IO () Source
Invoked when an executor has exited/terminated. Note that any
tasks running will have Lost status updates automagically
generated.
errorMessage :: a -> SchedulerDriver -> ByteString -> IO () Source
Invoked when there is an unrecoverable error in the scheduler or scheduler driver. The driver will be aborted BEFORE invoking this callback.
data SchedulerDriver Source
Type representing the connection from a scheduler to Mesos. This handle is used both to manage the scheduler's lifecycle (start it, stop it, or wait for it to finish) and to interact with Mesos (e.g., launch tasks, kill tasks, etc.).
Instances
Managing framework actions with a SchedulerDriver
SchedulerDriver lifecycle management
Arguments
| :: ToScheduler a | |
| => a | |
| -> FrameworkInfo | |
| -> ByteString |
|
| -> Maybe Credential | |
| -> (SchedulerDriver -> IO b) | |
| -> IO b |
start :: SchedulerDriver -> IO Status Source
Starts the scheduler driver. This needs to be called before any other driver calls are made.
Arguments
| :: SchedulerDriver | |
| -> Bool | should failover? |
| -> IO Status |
Stops the scheduler driver. If the failover flag is set to
false then it is expected that this framework will never
reconnect to Mesos and all of its executors and tasks can be
terminated. Otherwise, all executors and tasks will remain
running (for some framework specific failover timeout) allowing the
scheduler to reconnect (possibly in the same process, or from a
different process, for example, on a different machine).
abort :: SchedulerDriver -> IO Status Source
Aborts the driver so that no more callbacks can be made to the
scheduler. The semantics of abort and stop have deliberately been
separated so that code can detect an aborted driver (i.e., via
the return status of await, see below), and
instantiate and start another driver if desired (from within the
same process). Note that stop is not automatically called
inside abort.
await :: SchedulerDriver -> IO Status Source
Waits for the driver to be stopped or aborted, possibly
*blocking the current thread indefinitely*. The return status of
this function can be used to determine if the driver was aborted
(see Status for more information).
Managing and interacting with tasks and resources
requestResources :: SchedulerDriver -> [Request] -> IO Status Source
Requests resources from Mesos. Any resources available are asynchronously offered to the framework via the resourceOffers callback.
launchTasks :: SchedulerDriver -> [OfferID] -> [TaskInfo] -> Filters -> IO Status Source
Launches the given set of tasks. Any resources remaining (i.e.,
not used by the tasks or their executors) will be considered
declined. The specified filters are applied on all unused
resources (see Filters for more information).
Available resources are aggregated when mutiple offers are
provided. Note that all offers must belong to the same slave.
Invoking this function with an empty collection of tasks declines
offers in their entirety (see declineOffer).
killTask :: SchedulerDriver -> TaskID -> IO Status Source
Kills the specified task. Note that attempting to kill a task is currently not reliable. If, for example, a scheduler fails over while it was attempting to kill a task it will need to retry in the future. Likewise, if unregistered / disconnected, the request will be dropped (these semantics may be changed in the future).
declineOffer :: SchedulerDriver -> OfferID -> Filters -> IO Status Source
Declines an offer in its entirety and applies the specified
filters on the resources (see Filters). Note that this can be done at any time, it is not
necessary to do this within the resourceOffers
callback.
reviveOffers :: SchedulerDriver -> IO Status Source
Removes all filters previously set by the framework (via
launchTasks). This enables the framework to receive offers from
those filtered slaves.
sendFrameworkMessage :: SchedulerDriver -> ExecutorID -> SlaveID -> ByteString -> IO Status Source
Sends a message from the framework to one of its executors. These messages are best effort; do not expect a framework message to be retransmitted in any reliable fashion.
reconcileTasks :: SchedulerDriver -> [TaskStatus] -> IO Status Source
Reconciliation of tasks causes the master to send status updates for tasks whose status differs from the status sent here.
Low-level lifecycle management
createDriver :: Scheduler -> FrameworkInfo -> ByteString -> Maybe Credential -> IO SchedulerDriver Source
destroyDriver :: SchedulerDriver -> IO () Source
createScheduler :: ToScheduler a => a -> IO Scheduler Source
destroyScheduler :: Scheduler -> IO () Source