Tracing/Debugging Facilities

Cloud Haskell provides a general purpose tracing mechanism, allowing a user supplied tracer process to receive messages when certain classes of system events occur. It's possible to use this facility to aid in debugging and/or perform other diagnostic tasks to a program at runtime.

Enabling Tracing

Throughout the lifecycle of a local node, the distributed-process runtime generates trace events, describing internal runtime activities such as the spawning and death of processes, message sending, delivery and so on. See the MxEvent type's documentation for a list of all the published event types, which correspond directly to the types of management events. Users can additionally publish custom trace events in the form of MxLog log messages or pass custom (i.e., completely user defined) event data using the traceMessage function.

All published traces are forwarded to a tracer process, which can be specified (and changed) at runtime using traceEnable. Some pre-defined tracer processes are provided for conveniently printing to stderr, a log file or the GHC eventlog.

If a tracer process crashes, no attempt is made to restart it.

Working with multiple tracer processes

The tracing facility only ever writes to a single tracer process. This invariant insulates the tracer controller and ensures a fast path for handling all trace events. This module provides facilities for layering trace handlers using Cloud Haskell's built-in delegation primitives.

The startTracer function wraps the registered tracer process with the supplied handler and also forwards trace events to the original tracer. The corresponding stopTracer function terminates tracer processes in reverse of the order in which they were started, and re-registers the previous tracer process.

Built in tracers

The built in tracers provide a simple logging facility that writes trace events out to either a log file, stderr or the GHC eventlog. These tracers can be configured using environment variables, or specified manually using the traceEnable function.

When a new local node is started, the contents of several environment variables are checked to determine which default tracer process is selected. If none of these variables is set, a no-op tracer process is installed, which effectively ignores all trace messages. Note that in this case, trace events are still generated and passed through the system. Only one default tracer will be chosen - the first that contains a (valid) value. These environment variables, in the order they're examined, are:

1. DISTRIBUTED_PROCESS_TRACE_FILE This is checked for a valid file path. If it exists and the file can be opened for writing, all trace output will be directed thence. If the supplied path is invalid, or the file is unavailable for writing, this tracer will not be selected.
2. DISTRIBUTED_PROCESS_TRACE_CONSOLE This is checked for any non-empty value. If set, then all trace output will be directed to the system logger process.
3. DISTRIBUTED_PROCESS_TRACE_EVENTLOG This is checked for any non-empty value. If set, all internal traces are written to the GHC eventlog.

By default, the built in tracers will ignore all trace events! In order to enable tracing the incoming MxEvent stream, the DISTRIBUTED_PROCESS_TRACE_FLAGS environment variable accepts the following flags, which enable tracing specific event types:

• p = trace the spawning of new processes
• d = trace the death of processes
• n = trace registration of names (i.e., named processes)
• u = trace un-registration of names (i.e., named processes)
• s = trace the sending of messages to other processes
• r = trace the receipt of messages from other processes
• l = trace node up/down events

Users of the simplelocalnet Cloud Haskell backend should also note that because the trace file option only supports trace output from a single node (so as to avoid interleaving), a file trace configured for the master node will prevent slaves from tracing to the file. They will need to fall back to the console or eventlog tracers instead, which can be accomplished by setting one of these environment variables as well, since the latter will only be selected on slaves (when the file tracer selection fails).

Support for writing to the eventlog requires specific intervention to work, without which, written traces are silently dropped/ignored and no output will be generated. The GHC eventlog documentation provides information about enabling, viewing and working with event traces at http://hackage.haskell.org/trac/ghc/wiki/EventLog.