transient-universe: fully composable remote execution for the creation of distributed systems

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

fully composable remote execution for the creation of distributed systems across Web clients and servers using sockets, websockets and HTTP. Web API compatible, map-reduce implementation. See http://github.com/transient-haskell/transient-stack/transient-universe.


[Skip to Readme]

Properties

Versions 0.2, 0.3, 0.3.1.2, 0.3.2, 0.3.2.1, 0.3.2.2, 0.3.2.3, 0.3.4, 0.3.5, 0.3.5.1, 0.4.0, 0.4.1, 0.4.4, 0.4.4.1, 0.4.5, 0.4.6, 0.4.6.1, 0.5.0.0, 0.6.0.0, 0.6.0.0, 0.6.0.1
Change log ChangeLog.md
Dependencies aeson, base (>4 && <5), base64-bytestring, bytestring, case-insensitive, containers, directory, filepath, ghcjs-base, ghcjs-prim, hashable, HTTP, iproute, mtl, network (>=2.8.0.0 && <3.0.0.0), network-info, network-uri, old-time, process, random, stm, TCache (>=0.12), text, time, transformers, transient (>=0.7.0.0), transient-universe, vector, websockets (>=0.12.7.1) [details]
License MIT
Author Alberto G. Corona
Maintainer agocorona@gmail.com
Category Control, Distributed Computing
Home page https://github.com/transient-haskell/transient-universe
Bug tracker https://github.com/transient-haskell/transient-universe/issues
Source repo head: git clone https://github.com/transient-haskell/transient-universe
Uploaded by AlbertoCorona at 2020-07-29T08:27:05Z

Modules

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees


Readme for transient-universe-0.6.0.0

[back to package description]

Universe logo

Hackage Stackage LTS Stackage Nightly Build Status Gitter

See the Wiki

transient-universe is the distributed computing extension of transient and uses transient primitives heavily for parsing, threading, event handling, exception handling, messaging etc. It support moving computations between Haskell closures in different computers in the network. Even among different architectures: Linux nodes can work with windows and browser nodes running haskell compiled with ghcjs.

The primitives that perform the moving of computations are called wormhole and teleport, the names express the semantics. Hence the name of the package.

All the nodes run the same program compiled for different architectures. It defines a Cloud computation (monad). It is a thin layer on top of transient with additional primitives and services that run a single program in one or many nodes.

Example:

import Transient.Base
import Transient.Move
import Control.Monad

main= keep . initNode $ inputNodes <|> mypPogram

myProgram :: Cloud ()
myProgram= do
    nodes <- local getNodes
    guard $ length nodes > 1
    let node2= nodes !! 1
    r <- runAt node2 . local $ waitEvents getLine
    localIO $ print r
    

This program will stream and print any text that you input in the console of the node 2.

To know how to initialize the nodes, see the section of the Tutorial

Browser integration

Browser nodes, running transient programs compiled with ghcjs are integrated with server nodes, using websockets for communication. Just compile the program with ghcjs and point the browser to http://server:port. The server nodes have a HTTP server that will send the compiled program to the browser.

Distributed Browser/server Widgets

Browser nodes can integrate a reactive client side library based in trasient (package axiom). These widgets can create widgets with HTML form elements and control the server nodes. A computation can move from browser to server and back despite the different architecture.

This program will obtain a string from the browser, will send it to the server, which will return three responses wich will be presented in the browser:

import Transient.Base
import Transient.Move
import Transient.Indeterminism
import GHCJS.HPlay.View

main= keep . initNode $ myProgram

myProgram :: Cloud ()
myProgram=  do
  name <- local . render $ getString Nothing `fire` OnChange
  r <- atRemote . local . choose . take 3 . repeat $ "hello "++ name
  local . render . rawHtml $ h1 r

See the package Axiom for instructions about how to compile and run this program.

Widgets with code running in browser and servers can compose with other widgets. A Browser node can gain access to many server nodes trough the server that delivered the web application.

These features can make transient ideal for client as well as server side-driven applications, whenever distribution and push-driven reactivity is necessary either in the servers or in the browser clients.

New

The last release add

Map-reduce

transient-universe implements map-reduce in the style of spark as a particular case. It is at the same time a hard test of the distributed primitives since it involves a complex choreography of movement of computations. It supports in memory operations and caching. Resilience (restart from the last checkpoint in case of failure) is not implemented but it is foreseen.

Look at this article

There is a runnable example: DistrbDataSets.hs that you can executed with:

runghc ./examples/DistrbDataSets.hs

It uses a number of simulated nodes to calculate the frequency of words in a long text.

Services

Services communicate two different transient applications. This allows to divide the running application in different independent tiers. No documentation is available yet. Sorry.

General distributed primitives

teleport is a primitive that translates computations back and forth reusing an already opened connection.

The connection is initiated by wormhole with another node. This can be done anywhere in a computation without breaking composability. As always, Everything is composable.

Both primitives support also streaming among nodes in an efficient way. It means that a remote call can return not just a single response, but many of them.

All the other distributed primitives: runAt, streamFrom clustered etc are rewritten in terms of these two.

How to run the ghcjs example:

See the distributed examples in the transient-examples repository

See this video to see this example running:

The test program run among other things, two copies of a widget that start, stop and display a counter that run in the server.

Documentation

The Wiki is more user oriented

My video sessions in livecoding.tv not intended as tutorials or presentations, but show some of the latest features running.

The articles are more technical:

These articles contain executable examples (not now, since the site no longer support the execution of Haskell snippets).

Future plans

The only way to improve it is using it. Please send me bugs and additional functionalities!

-I plan to improve map-reduce to create a viable platform for serious data analysis and machine learning using haskell. It will have a web notebook running in the browser.

-Create services and examples for general Web applications with distributed servers and create services for them