hw-json: Memory efficient JSON parser

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

'ghc-options: -O2' is rarely needed. Check that it is giving a real benefit and not just imposing longer compile times on your users.

Memory efficient JSON parser. Please see README.md

[Skip to Readme]

Properties

Versions	0.0.0.1, 0.0.0.2, 0.0.0.3, 0.0.0.4, 0.1.0.0, 0.2.0.0, 0.2.0.1, 0.2.0.2, 0.2.0.3, 0.2.0.4, 0.3.0.0, 0.4.0.0, 0.5.0.0, 0.6.0.0, 0.7.0.1, 0.8.0.0, 0.8.1.0, 0.9.0.0, 0.9.0.1, 1.0.0.0, 1.0.0.0, 1.0.0.1, 1.0.0.2, 1.1.0.0, 1.1.0.1, 1.1.0.2, 1.2.0.0, 1.2.0.1, 1.2.0.2, 1.3.0.0, 1.3.0.1, 1.3.1.0, 1.3.1.1, 1.3.2.0, 1.3.2.1, 1.3.2.2, 1.3.2.3, 1.3.2.4, 1.3.2.5, 1.3.3.0
Change log	None available
Dependencies	ansi-wl-pprint (>=0.6.8.2 && <0.7), array (>=0.5 && <0.6), attoparsec (>=0.13 && <0.14), base (>=4 && <5), bits-extra (>=0.0.1 && <0.1), bytestring (>=0.10.6 && <0.11), dlist (>=0.8 && <0.9), hw-balancedparens (>=0.2.0.1 && <0.3), hw-bits (>=0.7.0.5 && <0.8), hw-json, hw-json-simd (>=0.1.0.1 && <0.2), hw-mquery (>=0.2.0.0 && <0.3), hw-parser (>=0.1 && <0.2), hw-prim (>=0.6.2.21 && <0.7), hw-rankselect (>=0.12.0.4 && <0.13), hw-rankselect-base (>=0.3.2.1 && <0.4), hw-simd (>=0.1.1.2 && <0.2), lens (>=4 && <5), mmap (>=0.5 && <0.6), optparse-applicative (>=0.14 && <0.15), semigroups (>=0.16 && <0.19), text (>=1.2 && <1.3), vector (>=0.12 && <0.13), word8 (>=0.1 && <0.2) [details]
License	BSD-3-Clause
Copyright	2016 - 2019 John Ky
Author	John Ky
Maintainer	newhoggy@gmail.com
Category	Data
Home page	http://github.com/haskell-works/hw-json#readme
Bug tracker	https://github.com/haskell-works/hw-json/issues
Source repo	head: git clone https://github.com/haskell-works/hw-json
Uploaded	by haskellworks at 2019-01-31T22:12:47Z

Modules

[Index] [Quick Jump]

HaskellWorks
- Data
  - HaskellWorks.Data.Json

Flags

Automatic Flags

Name	Description	Default
bmi2	Enable bmi2 instruction set	Disabled
sse42	Enable sse4.2 instruction set	Disabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

hw-json-1.0.0.0.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

newhoggy, haskellworks

For package maintainers and hackage trustees

edit package information

Readme for hw-json-1.0.0.0

[back to package description]

hw-json

hw-json is a succinct JSON parsing library. It uses succinct data-structures to allow traversal of large JSON strings with minimal memory overhead.

For an example, see app/Main.hs

Prerequisites

Install haskell-stack.
Install hlint (eg. stack install hlint)

Memory benchmark

Parsing large Json files in Scala with Argonaut

      S0U       EU           OU       MU     CCSU CMD
--------- --------- ----------- -------- -------- ---------------------------------------------------------------
      0.0  80,526.3    76,163.6 72,338.6 13,058.6 sbt console
      0.0 536,660.4    76,163.6 72,338.6 13,058.6 import java.io._, argonaut._, Argonaut._
      0.0 552,389.1    76,163.6 72,338.6 13,058.6 val file = new File("/Users/jky/Downloads/78mbs.json"
      0.0 634,066.5    76,163.6 72,338.6 13,058.6 val array = new Array[Byte](file.length.asInstanceOf[Int])
      0.0 644,552.3    76,163.6 72,338.6 13,058.6 val is = new FileInputStream("/Users/jky/Downloads/78mbs.json")
      0.0 655,038.1    76,163.6 72,338.6 13,058.6 is.read(array)
294,976.0 160,159.7 1,100,365.0 79,310.8 13,748.1 val json = new String(array)
285,182.9 146,392.6 1,956,264.5 82,679.8 14,099.6 val data = Parse.parse(json)
                    ***********

Parsing large Json files in Haskell with Aeson

Mem (MB) CMD
-------- ---------------------------------------------------------
      54 import Control.DeepSeq
      64 import Data.Aeson
      64 import qualified Data.ByteString.Lazy as BSL
      66 json78m <- BSL.readFile "../data/78mb.json"
     167 let !x = deepseq json78m json78m
    1207 let !y = decode json78m :: Maybe Value

Parsing large Json files in Haskell with hw-json

Mem (MB) CMD
-------- ---------------------------------------------------------
      57 import           Foreign
      58 import           Control.Monad
      59 import qualified Data.ByteString as BS
      76 import qualified Data.Vector.Storable as DVS
      77 import           Data.Word
      77 import           HaskellWorks.Data.BalancedParens.Simple
      77 import           HaskellWorks.Data.Bits.BitShown
      77 import           HaskellWorks.Data.FromForeignRegion
      81 import           HaskellWorks.Data.Json.Succinct.Cursor
      82 import           HaskellWorks.Diagnostics
      83 import           System.IO.MMap
      88 (fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "../data/78mbs.json" ReadOnly Nothing
     276 cursor <- measure (fromForeignRegion (fptr, offset, size) :: JsonCursor BS.ByteString (DVS.Vector Word64) (SimpleBalancedParens (DVS.Vector Word64)))

Examples

Performance example

(This section out of date)

import Foreign
import qualified Data.Vector.Storable as DVS
import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BSI
import System.IO.MMap
import Data.Word
import System.CPUTime
(fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "/Users/jky/Downloads/78mbs.json" ReadOnly Nothing
cursor <- measure (fromForeignRegion (fptr, offset, size) :: JsonCursor BS.ByteString (DVS.Vector Word64) (SimpleBalancedParens (DVS.Vector Word64)))
let !bs = BSI.fromForeignPtr (castForeignPtr fptr) offset size
x <- measure $ jsonBsToInterestBs bs
let !y = BS.concat (unescape' "" [bs])

import Foreign
import qualified Data.Vector.Storable as DVS
import qualified Data.ByteString as BS
import qualified Data.ByteString.Internal as BSI
import System.IO.MMap
import Data.Word
import System.CPUTime
(fptr :: ForeignPtr Word8, offset, size) <- mmapFileForeignPtr "/Users/jky/Downloads/part40.json" ReadOnly Nothing
let !bs = BSI.fromForeignPtr (castForeignPtr fptr) offset size
x <- measure $ BS.concat $ blankedJsonToInterestBits $ blankJson [bs]
x <- measure $ jsonBsToInterestBs bs

jsonTokenAt $ J.nextSibling $ J.firstChild $ J.nextSibling $ J.firstChild $ J.firstChild  cursor

import qualified Data.ByteString                             as BS
import           Data.String
import qualified Data.Vector.Storable                        as DVS
import           Data.Word
import           HaskellWorks.Data.Bits.BitShow
import           HaskellWorks.Data.Bits.BitShown
import           HaskellWorks.Data.FromForeignRegion
import           HaskellWorks.Data.Json.Succinct.Cursor      as C
import           HaskellWorks.Data.Json.Token
import           HaskellWorks.Data.BalancedParens.Simple
import           HaskellWorks.Data.RankSelect.Base.Rank0
import           HaskellWorks.Data.RankSelect.Base.Rank1
import           HaskellWorks.Data.RankSelect.Base.Select1
import           HaskellWorks.Data.RankSelect.Poppy512
import qualified HaskellWorks.Data.TreeCursor as TC
import           System.IO.MMap
let fc = TC.firstChild
let ns = TC.nextSibling
let pn = TC.parent
let ss = TC.subtreeSize
let cursor = "[null, {\"field\": 1}]" :: JsonCursor BS.ByteString (DVS.Vector Word64) (SimpleBalancedParens (DVS.Vector Word64))
cursor
fc cursor
(fc >=> ns) cursor

Querying example

import Control.Monad
import Data.Function
import Data.List
import HaskellWorks.Data.Json.Backend.Standard.Load
import HaskellWorks.Data.Json.LoadCursor
import HaskellWorks.Data.Json.PartialValue
import HaskellWorks.Data.MQuery
import HaskellWorks.Data.MQuery.Micro
import HaskellWorks.Data.MQuery.Row
import HaskellWorks.Diagnostics

import qualified Data.DList as DL

!cursor <- loadJsonPartial "../data/78mb.json"
!cursor <- loadJsonWithIndex "../data/78mb.json"
!cursor <- loadJson "../data/78mb.json"
!cursor <- loadJsonWithCsPoppyIndex "../data/78mb.json"
let !json = jsonPartialJsonValueAt cursor
let q = MQuery (DL.singleton json)

measureIO $ putPretty $ q >>= item & limit 10
measureIO $ putPretty $ q >>= item & page 10 1
measureIO $ putPretty $ q >>= item >>= hasKV "founded_year" (JsonPartialNumber 2005) & limit 10
measureIO $ putPretty $ q >>= item >>= entry
measureIO $ putPretty $ q >>= item >>= entry >>= named "name" & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code")
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code") & onList (uniq . sort)
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

import           Control.Monad
import qualified Data.DList as DL
import           Data.Function
import           Data.List
import           HaskellWorks.Data.Json.LoadCursor
import           HaskellWorks.Data.Micro
import           HaskellWorks.Data.MQuery
import           HaskellWorks.Data.Json.LightJson
import           HaskellWorks.Data.Row
import           HaskellWorks.Diagnostics

!cursor <- loadJsonPartial "../data/78mb.json"
!cursor <- loadJsonWithIndex "../data/78mb.json"
!cursor <- loadJson "../data/78mb.json"
!cursor <- loadJsonWithCsPoppyIndex "../data/78mb.json"
let !json = lightJsonAt cursor
let q = MQuery (DL.singleton json)

measureIO $ putPretty $ q >>= item & limit 10
measureIO $ putPretty $ q >>= item & page 10 1
measureIO $ putPretty $ q >>= item >>= entry
measureIO $ putPretty $ q >>= item >>= entry >>= named "name" & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString)
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString) & onList (uniq . sort)
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
measureIO $ putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

Decoding

Line separated base 64 encoded gzipped json

while read in; do echo "$in" | base64 --decode | gunzip; echo ""; done < file.lgz > firehose.json

Profiling with stack traces

mafia build -p
cabal repl --ghc-options='-fexternal-interpreter -prof'

import HaskellWorks.Data.Succinct.BalancedParens
import HaskellWorks.Data.Succinct.RankSelect.Binary.Poppy512
import HaskellWorks.Data.Positioning
import qualified Data.Vector.Storable as DVS
import HaskellWorks.Data.IndexedSeq
(jsonBS, jsonIb, jsonBp) <- loadRawWithIndex "firehose.json"
let bp1 = SimpleBalancedParens jsonBp
let bp2 = SimpleBalancedParens (makePoppy512 jsonBp)
let bp3 = makePoppy512 jsonBp

References

Special mentions

Sydney Paper Club

hw-json: Memory efficient JSON parser

Properties

Modules

Flags

Downloads

Maintainer's Corner

Readme for hw-json-1.0.0.0

hw-json

Prerequisites

Memory benchmark

Parsing large Json files in Scala with Argonaut

Parsing large Json files in Haskell with Aeson

Parsing large Json files in Haskell with hw-json

Examples

Performance example

Navigation example

Querying example

Decoding

Line separated base 64 encoded gzipped json

Profiling with stack traces

References

Special mentions