Main module of the stochastic gradient descent (SGD) library.

SGD is a method for optimizing a global objective function defined as a sum of smaller, differentiable functions. The individual component functions share the same set of parameters, represented by the ParamSet class.

To perform SGD, the gradients of the individual functions need to be determined. This can be done manually or automatically, using one of the automatic differentiation libraries (ad, backprop) available in Haskell.

For instance, let's say we have a list of functions defined as:

funs = [\x -> 0.3*x^2, \x -> -2*x, const 3, sin]

The global objective is then defined as:

objective x = sum $ map ($x) funs

We can manually determine the individual derivatives:

derivs = [\x -> 0.6*x, const (-2), const 0, cos]

or use an automatic differentiation library, for instance:

import qualified Numeric.AD as AD
derivs = map
  (\k -> AD.diff (funs !! k))
  [0..length funs-1]

Finally, run allows to approach a (potentially local) minimum of the global objective function:

>>> run (momentum def id) (take 10000 $ cycle derivs) 0.0
4.180177042912455

where:

• (take 10000 $ cycle derivs) is the stream of training examples
• (momentum def id) is the selected SGD variant (momentum), supplied with the default configuration (def) and the function (id) for calculating the gradient from a training example
• 0.0 is the initial parameter value