hq

hq is a JSON processor inspired by jq. It queries JSON with optics and rewrites values with transformations, streaming input so large documents never load fully into memory.

Usage

hq [OPTIONS] COMMAND OPTIC [ARGS] < input.json

hq v0.1.2

Usage: hq [-f|--file FILE] [-r|--raw] [-c|--compact] [-j|--join] COMMAND

  Query JSON using optics

Available options:
  -f,--file FILE           Input JSON file, or '-' for stdin
  -r,--raw                 Print strings without JSON quotes
  -c,--compact             Print compact JSON
  -j,--join                Print without separators
  -h,--help                Show this help text

Available commands:
  fold                     
  preview                  
  set                      
  over                     
  delete

Commands

Optics are used with different queries:

preview accepts any optic and prints its first focus in document order, stopping input there; it is fold capped at one value. set takes a JSON literal and is over with a constant transformation. over fails the whole query when a focused value rejects the transformation (adding to a string, for example). Deleting a missing field, or an empty focus, leaves the input unchanged.

Input may hold any number of whitespace-separated JSON values (JSON-lines style); every command except preview processes each top-level value in turn.

Optic language

Optics select and traverse JSON values. Optics are composed with .; each component focuses on a new set of values. Spaces around . are optional, so @a.each.@b means @a . each . @b. Parentheses group explicitly: (@a . each) . @b.

An optic may focus on nothing at all (a missing field, a value of the wrong shape, an out-of-bounds ix); such focuses simply select no values. Rewriting through keys renames object members.

filter OPTIC TRANSFORMATION

Keep the input when TRANSFORMATION maps some value focused by OPTIC to true, and focus on nothing otherwise. The transformation must produce a boolean.

Single atoms stay bare:

filter @age == 30

Anything longer takes its own paren group per side:

filter (each . @age) (== 30)

A . after a bare optic starts an outer composition, so a dotted optic inside filter needs its sides parenthesized:

each . filter (@tags . each) (== "x")

Examples:

• @name : Focus the "name" field.
• @users . each . @name : Focus the "name" field of every element in "users".
• @users . each . @age : Focus the "age" field of every element in "users".
• each . _Number : Focus every number inside an array.
• @a . ix 1 : Focus the second element of the array in "a".

Transformations

Transformations describe how a focused value is rewritten. They apply to one value at a time and compose right-to-left with .: in A . B, B runs first and A runs over its result. So + 1 . * 2 doubles first, then adds one.

Operators bind tightest at the atoms, then ., then and/&&, then xor/^^, then or/|| loosest. Parentheses override: (== "x") or (== "hi").

Numbers (input and output are numbers)

Comparisons (input is a number, output is a boolean)

Strings (input is a string)

stripPrefix and stripSuffix leave the value unchanged when the affix is absent. The is* tests yield a boolean.

Arrays (input is an array)

Objects (input is an object)

Any value

Equality examples: == 30, == "hi", == true. set is over with const.

Booleans (inputs are booleans, outputs are booleans)

For example:

hq over '@users . each . @age' '+ 1'

increments every user's age, and

hq over '@users . each . @name' 'stripPrefix "dr. "'

strips the prefix from every user's name when present.

Static checks

Queries are checked before running. Steps that cannot work are rejected up front instead of failing mid-stream:

• / 0 is rejected: division by zero.
• A transformation applied where it cannot fit is rejected: over 'each._Number' '++ "x"' fails because the optic focuses numbers and the transformation needs a string. Optics without a pinned shape (@field, each, ix) focus values of unknown type and accept anything; only prisms (and keys) narrow the focus, so only provable mismatches are reported.
• filter predicates must produce booleans, and accept their sub-optic's focus type.

Examples

echo '{"name":"ada","score":96}' | hq fold '@name'
# "ada"

echo '{"name":"ada"}' | hq fold '@name' -r
# ada (raw: strings print without quotes)

echo '{"a":[1,2,3]}' | hq fold '@a . each'
# 1, 2, 3 (one per line)

echo '[{"a":90},{"a":96}]' | hq fold 'each . filter (@a) (== 90)' -c
# [{"a":90}]

echo '{"a":1}' | hq over '@a' '+ 1' -c
# {"a":2}

echo '{"a":1,"b":2}' | hq set '@b' '3' -c
# {"a":1,"b":3}

echo '{"a":1,"b":2}' | hq delete '@b' -c
# {"a":1}

# nix
nix build . --json \
  | hq -r fold 'each . @outputs . values' \
  | cachix push my-cachix-cache

# Kubernetes
kubectl get pods -o json \
  | hq -r fold '@items . each . @metadata . @name'

# ffmpeg
ffprobe -v quiet -print_format json -show_streams video.mp4 \
  | hq -r fold 'each . @codec_name'

# Github
curl -s https://api.github.com/repos/damianfral/hq \
  | hq -r fold '@topics.each'

Performance

On the 100 MB hq-bench-data benchmark (bench/bench.sh), fold runs about 3x faster than jq, while over/set/delete perform around 1.2x faster.

hq streams the input without materializing the document, keeping peak RSS at ~14MB versus ~600 MB for jq.