inf-backprop-0.1.0.0: Automatic differentiation and backpropagation.
Copyright(C) 2023 Alexey Tochin
LicenseBSD3 (see the file LICENSE)
MaintainerAlexey Tochin <Alexey.Tochin@gmail.com>
Safe HaskellSafe-Inferred
LanguageHaskell2010
Extensions
  • UndecidableInstances
  • MonoLocalBinds
  • ScopedTypeVariables
  • TypeFamilies
  • GADTs
  • GADTSyntax
  • ConstraintKinds
  • InstanceSigs
  • DeriveFunctor
  • TypeSynonymInstances
  • FlexibleContexts
  • FlexibleInstances
  • ConstrainedClassMethods
  • MultiParamTypeClasses
  • KindSignatures
  • TupleSections
  • RankNTypes
  • ExplicitNamespaces
  • ExplicitForAll

InfBackprop.Common

Description

Provides base types and methods for backpropagation category morphism.

Synopsis

Basic

data Backprop cat input output Source #

Backprop morphism. Base type for an infinitely differentiable object. It depends on categorical type cat that is mostly common (->), see BackpropFunc which by it's definition is equivalent to

data BackpropFunc input output = forall cache. MkBackpropFunc {
 call     :: input -> output,
 forward  :: BackpropFunc input (output, cache),
 backward :: BackpropFunc (output, cache) input
}

The diagram below illustrates the how it works for the first derivative. Consider the role of function f in the derivative of the composition g(f(h(...))).

  h        ·                  f                   ·        g
           ·                                      ·
           ·               forward                ·
           · --- input  >-----+-----> output >--- ·
           ·                  V                   ·
 ...       ·                  |                   ·       ...
           ·                  | cache             ·
           ·                  |                   ·
           ·                  V                   ·
           · --< dInput <-----+-----< dOutput <-- ·
           ·               backward               ·

Notice that forward and backward are of type BackpropFunc but not (->). This is needed for further differentiation. However for the first derivative this difference can be ignored.

The return type of forward contains additional term cache. It is needed to save and transfer data calculated in the forward step to the backward step for reuse. See an example in

Differentiation with logging section .

Remark

Expand

Mathematically speaking we have to distinguish the types for forward and for backward methods because the second acts on the cotangent bundle. However, for simplicity and due to technical reasons we identify the types input and dInput as well as output and dOutput which is enough for our purposes because these types are usually real numbers or arrays of real numbers.

Constructors

forall cache. MkBackprop (cat input output) (Backprop cat input (output, cache)) (Backprop cat (output, cache) input) 

Instances

Instances details
(Isomorphism cat, CatBiFunctor (,) cat) => Category (Backprop cat :: Type -> Type -> Type) Source # 
Instance details

Defined in InfBackprop.Common

Methods

id :: forall (a :: k). Backprop cat a a #

(.) :: forall (b :: k) (c :: k) (a :: k). Backprop cat b c -> Backprop cat a b -> Backprop cat a c #

(Isomorphism cat, CatBiFunctor (,) cat) => CatBiFunctor (,) (Backprop cat) Source # 
Instance details

Defined in InfBackprop.Common

Methods

(***) :: Backprop cat a1 b1 -> Backprop cat a2 b2 -> Backprop cat (a1, a2) (b1, b2) Source #

first :: Backprop cat a b -> Backprop cat (a, c) (b, c) Source #

second :: Backprop cat a b -> Backprop cat (c, a) (c, b) Source #

(Isomorphism cat, CatBiFunctor (,) cat) => Isomorphism (Backprop cat) Source # 
Instance details

Defined in InfBackprop.Common

Methods

iso :: IsomorphicTo a b => Backprop cat a b Source #

call :: Backprop cat input output -> cat input output Source #

Simple internal category object extraction.

forward :: Backprop cat input output -> Backprop cat input (output, cache) Source #

Returns forward category. In the case cat = (->), the method coincides with Backprop cat input output itself but the output contains an additional data term cache with some calculation result that can be reused on in backward.

backward :: Backprop cat input output -> Backprop cat (output, cache) input Source #

Returns backward category. In the case cat = (->), the method takes the additional data term cache that is calculated in forward.

class Distributive x => StartBackprop cat x Source #

Interface for categories cat and value types x that support starting the backpropagation. For example for (->) and Float we are able to start the backpropagation like f(g(x)) -> 1 · f'(g(x)) · ... because f is a Float valued (scalar) function. Calculating Jacobians is not currently implemented.

Minimal complete definition

startBackprop

Instances

Instances details
(Distributive x, Monad m) => StartBackprop (Kleisli m) x Source # 
Instance details

Defined in InfBackprop.Common

Distributive x => StartBackprop (->) x Source # 
Instance details

Defined in InfBackprop.Common

Methods

startBackprop :: Backprop (->) x x Source #

startBackprop :: StartBackprop cat x => Backprop cat x x Source #

Morphism that connects forward and backward chain. Usually it is just 1 that is supposed to be multiplied on the derivative of the top function.

forwardBackward Source #

Arguments

:: (Isomorphism cat, CatBiFunctor (,) cat) 
=> Backprop cat y y

backprop morphism between y and dy that is inferred after the forward step for f and before the backward step for f

-> Backprop cat x y

some backprop morphism f between x and y

-> Backprop cat x x

the output backprop morphism from x to dx that is the composition.

Implementation of the process illustrated in the diagram. The first argument is a backprop morphism y -> dy The second argument is a backprop morphism x -> y The output is the backprop x -> dx build according the diagram

numba :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat y) => Backprop cat x y -> Backprop cat x x Source #

Backpropagation derivative in terms of backprop morphisms.

numbaN :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat x) => Natural -> Backprop cat x x -> Backprop cat x x Source #

Backpropagation ns derivative in terms of backprop morphisms.

derivative :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat y) => Backprop cat x y -> cat x x Source #

Backpropagation derivative as categorical object. If cat is (->) the output is simply a function.

Examples of usage

Expand
>>> import InfBackprop (sin)
>>> import Prelude (Float)
>>> derivative sin (0 :: Float)
1.0

derivativeN :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat x) => Natural -> Backprop cat x x -> cat x x Source #

Backpropagation derivative of order n as categorical object. If cat is (->) the output is simply a function.

Examples of usage

Expand
>>> import InfBackprop (pow, const)
>>> import Prelude (Float, fmap)
>>> myFunc = (pow 2) :: Backprop (->) Float Float
>>> fmap (derivativeN 0 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[9.0,4.0,1.0,0.0,1.0,4.0,9.0]
>>> fmap (derivativeN 1 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[-6.0,-4.0,-2.0,0.0,2.0,4.0,6.0]
>>> fmap (derivativeN 2 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[2.0,2.0,2.0,2.0,2.0,2.0,2.0]
>>> fmap (derivativeN 3 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0]

Differentiable functions

type BackpropFunc = Backprop (->) Source #

Infinitely differentiable function. The definition of the type synonym is equivalent to

data BackpropFunc input output = forall cache. MkBackpropFunc {
   call     :: input -> output,
   forward  :: BackpropFunc input (output, cache),
   backward :: BackpropFunc (output, cache) input
 }

See Backprop for details.

Examples of usage

Expand
>>> import Prelude (fmap, Float)
>>> import InfBackprop (pow, call, derivative)
>>> myFunc = pow 2 :: BackpropFunc Float Float
>>> f = call myFunc :: Float -> Float
>>> fmap f [-3, -2, -1, 0, 1, 2, 3]
[9.0,4.0,1.0,0.0,1.0,4.0,9.0]
>>> df = derivative myFunc :: Float -> Float
>>> fmap df [-3, -2, -1, 0, 1, 2, 3]
[-6.0,-4.0,-2.0,0.0,2.0,4.0,6.0]

const :: forall c x. (Additive c, Additive x) => c -> BackpropFunc x c Source #

Infinitely differentiable constant function.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative, derivativeN)
>>> call (const 5) ()
5
>>> derivative (const (5 :: Float)) 42
0
>>> derivativeN 2 (const (5 :: Float)) 42
0.0

Differentiable monadic functions

pureBackprop :: forall a b m. Monad m => Backprop (->) a b -> Backprop (Kleisli m) a b Source #

Lifts a backprop function morphism to the corresponding pure Kleisli morphism.