A light-weight call-stack profiler for GHC!

GHC Stack Profiler

⚠️ Warning: This package is experimental. It is versioned according to the PVP. Breaking changes should be expected and no effort will be made to avoid major version bumps.

⚠️ Warning: Due to a bug in GHC, copying the call-stack may cause a segfault at runtime in applications built with GHC 9.14.1 and older. If you use GHC Stack Profiler in production, you should build your application with GHC 9.14.2 or newer.

GHC Stack Profiler periodically samples the GHC runtime call-stack and writes these samples to the eventlog. These eventlogs can be used in two ways:

• ghc-stack-profiler-speedscope can be used to export call-stack profiles to speedscope.
• eventlog-live-otlp can stream call-stack profiles, in real-time, to any observability platform that supports the OpenTelemetry protocol, such as Grafana Cloud.

Unlike GHC's built-in cost-centre stack profiler, GHC Stack Profiler does not require you to rebuild your program with profiling support and has virtually no overhead when it's not running. (See Benchmarks.)

The following is a screenshot of speedscope that shows a call-stack profile of Agda version 2.8.0.1 checking the standard library created using GHC Stack Profiler:

Table of Contents

• Getting Started
  • Instrument your application with GHC Stack Profiler
  • GHC Stack Profiler with Speedscope
  • GHC Stack Profiler with Eventlog Live – Real-Time Call-Stack Profiles
  • GHC Stack Profiler with Eventlog Socket – Dynamic Control
• Benchmarks
  • Benchmark: Agda 2.8.0.1 checking the standard library
  • Benchmark: GHC 10.1 loading Cabal-syntax

Getting Started

Let's get GHC Stack Profiler working with your application, which we'll conveniently call your-application.

In the first two sections, we'll instrument your application with GHC Stack Profiler and visualise the call-stack profile of a completed run using speedscope. In the last two sections, we'll add Eventlog Live and Eventlog Socket to visualise your application's call-stack profiles in real-time and control GHC Stack Profiler from your observability dashboard.

Instrument your application with GHC Stack Profiler

To instrument your application with GHC Stack Profiler, you need to make four changes:

1. Add ghc-stack-profiler to the build-depends for your application:

     executable your-application
       ...
   
       build-depends:
         ...
   +     , ghc-stack-profiler ==0.5.0.0
   

   ⚠️ Warning: If you're using ghc-stack-profiler-speedscope, eventlog-live-otlp, or any other program that processes the eventlog produced by ghc-stack-profiler, it is important that both are built with the same version of ghc-stack-profiler-core.

2. Build your application with support for RTS options and the threaded runtime.

   Add the following to the executable section of your application:

     executable your-application
       ...
   
   +   ghc-options: -rtsopts
   +   ghc-options: -threaded
   

   The -rtsopts flag enables the RTS options for your application. This allows us to enable the eventlog at runtime and enable various kinds of profiling. Setting this option may pose a security risk. If this is a concern, you can set all the required RTS options at compile time using -with-rtsopts.

   The -threaded flag builds your application with the threaded RTS.

3. Instrument your main function:

     module Main where
     ...
   
   + import GHC.Stack.Profiler (withProfilerFromEnv)
   
     main :: IO ()
     main =
   +   withProfilerFromEnv $
         ...
   

   ℹ️ Tip: If you prefer not to configure your program from the environment, the GHC.Stack.Profiler exposes a variety of function that instrument your program.

   ℹ️ Tip: You can use the annotateStackIO functions from ghc-stack-annotations to push annotation frames onto the call-stack at runtime. These annotation frames are visible in call-stack profiles captured by GHC Stack Profiler. See Better Haskell stack traces via user annotations.

4. Build your application and its dependencies with info table maps.

   Let's do this in two steps:

   1. To build your application and its dependencies with info table maps, you must ensure that they are built with the -finfo-table-map and -fdistinct-constructor-tables GHC options.

      The easiest way to do this is to add the following to your cabal.project file:

      package *
        ghc-options:
          -finfo-table-map
          -fdistinct-constructor-tables
      

      There is currently no easy way to pass GHC options to all packages when using cabal install. As a workaround, you can add a cabal.project file to a source distribution and install from there.

      If you run GHC Stack Profiler with your application built this way, you will get detailed information for all the symbols defined in your application and most symbols defined in your dependencies. However, you will see some unresolved info tables, which will show as numbers, e.g., 0x100000000. These are symbols that are either built into GHC or defined in the boot libraries that came with GHC, such as base. The boot packages are never rebuilt by Cabal and are unaffected by the package * stanza.

   2. To build the GHC and the boot libraries with info table maps, you must build GHC with the +ipe flavour.

      The easiest way to do this is using ghcup. Some variant of the following command may work for you:

      ghcup compile ghc -j0 -b 9.10.3 -v 9.10.3 -f perf+ipe -o '%v-ipe' --
      

      You may need to pass the appropriate configure flags for your platform. See Building and Porting GHC.

   Once you have a version of GHC built with the +ipe flavour and rebuilt application, you should no longer see unresolved info tables.

GHC Stack Profiler with Speedscope

If you have instrumented your application, you can run it with GHC Stack Profiler and export a call-stack profile to the speedscope format:

# Configure GHC Stack Profiler
export GHC_STACK_PROFILER="ON" # or any other non-empty value
export GHC_STACK_PROFILER_SAMPLE_INTERVAL="10" # milliseconds

# Start your application
./your-application               \
    +RTS                         \
    -l                           \
    -olyour-application.eventlog \
    -RTS

# Export the eventlog to speedscope
ghc-stack-profiler-speedscope \
    your-application.eventlog \
    your-application.json

To view your call-stack profile, open speedscope and load your-application.json.

The ghc-stack-profiler-speedscope program has several options that control the speedscope profile:

• You can restrict your profile to the section between start and end markers (using --start/--end), which you can emit from your application using traceMarkerIO.

  Let's say your application has to do some setup and cleanup, but you're only interested in profiling The Big Chore. If you instrument your application as follows and call ghc-stack-profiler-speedscope with --start=START and --end=END, your profile will only include samples from The Big Chore:

  main = do
    doSomeSetup           -- Not included in profile.
    traceMarkerIO "START" -- Start marker.
    doTheBigChore         -- Included in profile.
    traceMarkerIO "END"   -- End marker.
    doSomeCleanup         -- Not included in profile.
  

  ℹ️ Tip: This applies a post-hoc filter, which means that GHC Stack Profiling will still be sampling during the setup and cleanup. If you want to sample only during The Big Chore, you can use either startProfiling/stopProfiling or the Eventlog Socket control commands.

• You can aggregate your application's profiles by thread or capability:

  • --per-thread: Group the profiles by thread. (Default.)
  • --per-capability: Group the profiles by capability.
  • --no-aggregation: Do not aggregate the profiles.

GHC Stack Profiler with Eventlog Live – Real-Time Call-Stack Profiles

If you have instrumented your application, you can run it with GHC Stack Profiler and Eventlog Live and stream call-stack profiles, in real-time, to any observability platform that supports the OpenTelemetry protocol, such as Grafana Cloud. For detailed instructions, see the section Eventlog Live with GHC Stack Profiler in the README for Eventlog Live.

The following shows real-time call-stack profiles visualised in Grafana:

GHC Stack Profiler with Eventlog Socket – Dynamic Control

When compiled with the +control feature flag, GHC Stack Profiler has built-in support for Eventlog Socket's control commands. This lets you dynamically start and stop profiling by writing the command to the eventlog socket. For a detailed explanation of control commands, see the section Control Commands in the README for Eventlog Socket.

If you are using Eventlog Live, you can use its control server to send the GHC Stack Profiler control commands via HTTP. This lets you control profiling from your observability dashboard, e.g., using the Start/Stop buttons at the bottom of the Grafana dashboard in the previous section. For detailed instructions, see the section Eventlog Live with Eventlog Socket in the README for Eventlog Live.

Benchmarks

This section discusses our benchmarks that measure the overhead of instrumenting and profiling your application with GHC Stack Profiler and GHC's built-in cost-centre profiler. Our conclusions:

• Instrumenting your application with GHC Stack Profiler has no measurable overhead.

  Running GHC Stack Profiler has about 2% overhead with no significant difference between the measured sample intervals.

• Instrumenting your application with the cost-centre profiler has around 50% overhead with no cost centres and around 100% overhead with late cost centres.

  Running the cost-centre profiler has an additional 2% overhead with no significant difference between the measured sample intervals.

Benchmark: Agda 2.8.0.1 checking the standard library

The benchmark measures Agda 2.8.0.1 checking the standard library:

# from within std-lib/ in the Agda repository
agda --build-library +RTS -N1

There are three classes of benchmarks:

• The baseline benchmark uses Agda with no modifications.

• The ghc-stack-profiler benchmarks use Agda instrumented with ghc-stack-profiler.

  (For details, see Instrument your application with GHC Stack Profiler.)

• The profiling benchmarks use Agda instrumented with cost-centre profiling, using the following cabal.project, where the value of profiling-detail taken from the benchmark name:

  profiling: True
  
  package *
    profiling-detail: none -- or late
  

  (The -p RTS option was used to the profiler and the -V RTS option was used to set the sample interval.)

The results are normalised as a percentage of the baseline benchmark which took, on average, 3 minutes and 55 seconds on an otherwise idle machine. The timings are the result of, on average, 10 runs excluding warm-up.

Benchmark: GHC 10.1 loading Cabal-syntax

The benchmark measures GHC 10.1 (9a442c9383) loading Cabal-syntax in interactive mode, using the GHC command obtained from hie-bios:

# from within libraries/Cabal/ in the GHC repository
hie-bios -v debug Cabal-syntax/src/Distribution/CabalSpecVersion.hs

There two classes of benchmarks:

• The ghc-stack-profiler benchmarks use GHC instrumented with ghc-stack-profiler.

  (For details, see Instrument your application with GHC Stack Profiler.)

• The profiling benchmarks use GHC instrumented with cost-centre profiling, using the default build flavour with the profiled_ghc flavour transformer, which builds GHC and its dependencies with profiling and adds late cost centres.

  (The -pj RTS option was used to enable the profiler and the -V RTS option was used to set the sample interval.)

The results are normalised as a percentage of the ghc-stack-profiler (instrumented only) benchmark which took, on average, 7 seconds. The timings are the result of, on average, 3 runs without warm-up on a noisy machine. The measurement that shows that sampling at a 10ms interval is slower than a 1ms interval is likely due to this noise. We did not include a baseline benchmark with an uninstrumented GHC, as there was no measurable overhead in the previous benchmark. We also did not include a profiling (instrumented only, profiling-detail: none) benchmark, as that would have required adding a new flavour transformer to GHC's build system.