hw-json: Memory efficient JSON parser

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

'ghc-options: -O2' is rarely needed. Check that it is giving a real benefit and not just imposing longer compile times on your users.

Memory efficient JSON parser. Please see README.md

[Skip to Readme]

Properties

Versions	0.0.0.1, 0.0.0.2, 0.0.0.3, 0.0.0.4, 0.1.0.0, 0.2.0.0, 0.2.0.1, 0.2.0.2, 0.2.0.3, 0.2.0.4, 0.3.0.0, 0.4.0.0, 0.5.0.0, 0.6.0.0, 0.7.0.1, 0.8.0.0, 0.8.1.0, 0.9.0.0, 0.9.0.1, 1.0.0.0, 1.0.0.1, 1.0.0.1, 1.0.0.2, 1.1.0.0, 1.1.0.1, 1.1.0.2, 1.2.0.0, 1.2.0.1, 1.2.0.2, 1.3.0.0, 1.3.0.1, 1.3.1.0, 1.3.1.1, 1.3.2.0, 1.3.2.1, 1.3.2.2, 1.3.2.3, 1.3.2.4, 1.3.2.5, 1.3.3.0
Change log	None available
Dependencies	ansi-wl-pprint (>=0.6.8.2 && <0.7), array (>=0.5 && <0.6), attoparsec (>=0.13 && <0.14), base (>=4 && <5), bits-extra (>=0.0.1 && <0.1), bytestring (>=0.10.6 && <0.11), dlist (>=0.8 && <0.9), hw-balancedparens (>=0.2.0.1 && <0.3), hw-bits (>=0.7.0.5 && <0.8), hw-json, hw-json-simd (>=0.1.0.1 && <0.2), hw-mquery (>=0.2.0.0 && <0.3), hw-parser (>=0.1 && <0.2), hw-prim (>=0.6.2.21 && <0.7), hw-rankselect (>=0.12.0.4 && <0.13), hw-rankselect-base (>=0.3.2.1 && <0.4), hw-simd (>=0.1.1.2 && <0.2), lens (>=4 && <5), mmap (>=0.5 && <0.6), optparse-applicative (>=0.14 && <0.15), semigroups (>=0.16 && <0.19), text (>=1.2 && <1.3), vector (>=0.12 && <0.13), word8 (>=0.1 && <0.2) [details]
License	BSD-3-Clause
Copyright	2016 - 2019 John Ky
Author	John Ky
Maintainer	newhoggy@gmail.com
Category	Data
Home page	http://github.com/haskell-works/hw-json#readme
Bug tracker	https://github.com/haskell-works/hw-json/issues
Source repo	head: git clone https://github.com/haskell-works/hw-json
Uploaded	by haskellworks at 2019-02-01T05:56:08Z

Modules

[Index] [Quick Jump]

HaskellWorks
- Data
  - HaskellWorks.Data.Json

Flags

Automatic Flags

Name	Description	Default
bmi2	Enable bmi2 instruction set	Disabled
sse42	Enable sse4.2 instruction set	Disabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

hw-json-1.0.0.1.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

newhoggy, haskellworks

For package maintainers and hackage trustees

edit package information

Readme for hw-json-1.0.0.1

[back to package description]

hw-json

hw-json is a succinct JSON parsing library.

It uses succinct data-structures to allow traversal of large JSON strings with minimal memory overhead.

For an example, see app/Main.hs

Prerequisites

cabal version 2.2 or later

Memory benchmark

Parsing large Json files in Scala with Argonaut

      S0U       EU           OU       MU     CCSU CMD
--------- --------- ----------- -------- -------- ---------------------------------------------------------------
      0.0  80,526.3    76,163.6 72,338.6 13,058.6 sbt console
      0.0 536,660.4    76,163.6 72,338.6 13,058.6 import java.io._, argonaut._, Argonaut._
      0.0 552,389.1    76,163.6 72,338.6 13,058.6 val file = new File("/Users/jky/Downloads/78mbs.json"
      0.0 634,066.5    76,163.6 72,338.6 13,058.6 val array = new Array[Byte](file.length.asInstanceOf[Int])
      0.0 644,552.3    76,163.6 72,338.6 13,058.6 val is = new FileInputStream("/Users/jky/Downloads/78mbs.json")
      0.0 655,038.1    76,163.6 72,338.6 13,058.6 is.read(array)
294,976.0 160,159.7 1,100,365.0 79,310.8 13,748.1 val json = new String(array)
285,182.9 146,392.6 1,956,264.5 82,679.8 14,099.6 val data = Parse.parse(json)
                    ***********

Parsing large Json files in Haskell with Aeson

-- CMD                                                     -- Mem (MB)
---------------------------------------------------------- -- --------
import Control.DeepSeq                                     --       94
import Data.Aeson                                          --      100
import qualified Data.ByteString.Lazy as BSL               --      104
bs <- BSL.readFile "../corpus/bench/hospitalisation.json"  --      105
let !x = deepseq bs bs                                     --      146
let !y = decode json78m :: Maybe Value                     --      669

Parsing large Json files in Haskell with hw-json

-- CMD                                                                -- Mem (MB)
--------------------------------------------------------------------- -- --------
import Foreign                                                        --       93
import Control.Monad                                                  --       95
import Data.Word                                                      --       96
import HaskellWorks.Data.BalancedParens.Simple                        --       97
import HaskellWorks.Data.Bits.BitShown                                --       98
import HaskellWorks.Data.FromForeignRegion                            --       99
import HaskellWorks.Data.Json.Backend.Standard.Cursor                 --      106
import System.IO.MMap                                                 --      109
import qualified Data.ByteString                              as BS   --      110
import qualified Data.Vector.Storable                         as DVS  --      111
import qualified HaskellWorks.Data.ByteString                 as BS   --      112
import qualified HaskellWorks.Data.Json.Backend.Standard.Fast as FAST --      114
bs <- BS.mmap "../corpus/bench/hospitalisation.json"                  --      115
let !cursor = FAST.makeCursor bs                                      --      203

Examples

import Control.Monad
import Data.String
import Data.Word
import HaskellWorks.Data.BalancedParens.Simple
import HaskellWorks.Data.Bits.BitShow
import HaskellWorks.Data.Bits.BitShown
import HaskellWorks.Data.FromForeignRegion
import HaskellWorks.Data.Json.Backend.Standard.Cursor
import HaskellWorks.Data.Json.Internal.Token.Types
import HaskellWorks.Data.RankSelect.Base.Rank0
import HaskellWorks.Data.RankSelect.Base.Rank1
import HaskellWorks.Data.RankSelect.Base.Select1
import HaskellWorks.Data.RankSelect.Poppy512
import System.IO.MMap

import qualified Data.ByteString                                as BS
import qualified Data.Vector.Storable                           as DVS
import qualified HaskellWorks.Data.Json.Backend.Standard.Cursor as C
import qualified HaskellWorks.Data.Json.Backend.Standard.Fast   as FAST
import qualified HaskellWorks.Data.TreeCursor                   as TC

let fc = TC.firstChild
let ns = TC.nextSibling
let pn = TC.parent
let ss = TC.subtreeSize
let cursor = FAST.makeCursor "[null, {\"field\": 1}]"
cursor
fc cursor
(fc >=> ns) cursor

Querying example

import Control.Monad
import Data.Function
import Data.List
import HaskellWorks.Data.Json.Backend.Standard.Load.Cursor
import HaskellWorks.Data.Json.Backend.Standard.Load.Partial
import HaskellWorks.Data.Json.Backend.Standard.Load.Raw
import HaskellWorks.Data.Json.PartialValue
import HaskellWorks.Data.MQuery
import HaskellWorks.Data.MQuery.Micro
import HaskellWorks.Data.MQuery.Row

import qualified Data.DList as DL

!cursor <- loadPartial "../corpus/bench/78mb.json"
!cursor <- loadCursorWithIndex "../corpus/bench/78mb.json"
!cursor <- loadCursor "../corpus/bench/78mb.json"
!cursor <- loadCursorWithCsPoppyIndex "../corpus/bench/78mb.json"
let !json = jsonPartialJsonValueAt cursor
let q = MQuery (DL.singleton json)

putPretty $ q >>= item & limit 10
putPretty $ q >>= item & page 10 1
putPretty $ q >>= item >>= hasKV "founded_year" (JsonPartialNumber 2005) & limit 10
putPretty $ q >>= item >>= entry
putPretty $ q >>= item >>= entry >>= named "name" & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code")
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code") & onList (uniq . sort)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

putPretty $ q >>= item & limit 10
putPretty $ q >>= item & page 10 1
putPretty $ q >>= item >>= entry
putPretty $ q >>= item >>= entry >>= named "name" & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString) & onList (uniq . sort)
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount") & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & limit 10
putPretty $ q >>= (item >=> entry >=> named "acquisition" >=> having (entry >=> named "price_currency_code" >=> asString >=> valueOf "USD") >=> entry >=> named "price_amount" >=> castAsInteger ) & aggregate sum

References

Special mentions

Sydney Paper Club

hw-json: Memory efficient JSON parser

Properties

Modules

Flags

Downloads

Maintainer's Corner

Readme for hw-json-1.0.0.1

hw-json

Prerequisites

Memory benchmark

Parsing large Json files in Scala with Argonaut

Parsing large Json files in Haskell with Aeson

Parsing large Json files in Haskell with hw-json

Examples

Navigation example

Querying example

References

Special mentions