Setdown

Line-based set manipulation from the command line.

Author: Robert Massaioli · Created in 2015

What is setdown?

Setdown treats text files as sets — one element per line — and lets you combine them with intersection, union, difference, and symmetric difference. You describe the operations once in a .setdown file (think of it like a Makefile for sets), and setdown resolves the whole dependency graph, computes every definition, and writes one result file per definition.

InternalStaff:   "admins.txt" \/ "developers.txt"
AllUsers:        InternalStaff \/ "contractors.txt"
ContractorsOnly: "contractors.txt" - InternalStaff

Run setdown in the directory containing that file and you get an output/ directory with InternalStaff.txt, AllUsers.txt, and ContractorsOnly.txt — each sorted and de-duplicated.

Input files don't need to be sorted, de-duplicated, or even sets to begin with; setdown normalizes them as it goes. And setdown is current-working-directory invariant: all paths inside a .setdown file are resolved relative to that file, not to wherever you happen to run the command from, so you can invoke it from anywhere in your project tree and get the same result.

Installation

Via nix-shell (quickest, no local setup required)

$ nix-shell -p haskellPackages.setdown
$ setdown --help

Via Hackage

stack install setdown

This works because setdown is on Hackage. To build from source instead — for example, to get the latest unreleased changes — see Building the code below.

Quick start

Every example below lives under examples/ in this repository — clone the repo and run any of them directly.

git clone https://github.com/robertmassaioli/setdown.git
cd setdown/examples/access-control
stack exec -- setdown

Set operations and precedence

Intersection, union, and symmetric difference are commutative (A op B is the same as B op A). Difference is not (A - B ≠ B - A).

Symmetric difference (>< or △) yields the elements that appear in exactly one of the two inputs — those in A but not B, plus those in B but not A. It is equivalent to (A - B) \/ (B - A) but computed in a single pass; see examples/symmetric-difference for both forms side by side.

definition: (A - B) \/ (C /\ D)
changedSubscribers: "january.txt" >< "february.txt"

There is no operator precedence — you must bracket nested expressions explicitly. Consider:

def: A /\ B \/ C

Should this parse as (A /\ B) \/ C or A /\ (B \/ C)? Substitute the empty set for B and the two readings diverge completely (E versus A /\ C). Because the difference is not cosmetic, setdown refuses to guess — an unbracketed expression like this is a parse error.

Language reference

Identifiers

Definition names may contain letters, digits, hyphens, and underscores: [a-zA-Z0-9_-]+. For example, mySet, result-2, and Final_Output are all valid; spaces and other punctuation are not permitted.

Definition ordering

Definitions may appear in any order in a .setdown file, and a definition may reference another that's defined later on. Setdown resolves all identifiers by name after parsing the whole file.

Circular definitions

Definitions must not form a cycle:

A: "file.txt" \/ B
B: A /\ "other.txt"

Setdown detects cycles like this and exits with an error before performing any operations — see examples/cycle-detection.

Comments

Add comments with a double-dash (--) through the end of the line, anywhere on the line:

-- This is a definition for A, created because we wanted to do X
A: "y.txt" - "z.txt"

B: (A \/ C) -- \/ D   This is still a comment and \/ D never happens

Full example

-- A is the intersection of the file b-1.out and the set B
A: "b-1.out" /\ B

-- B is the union of the file a-1.out and a-2.out
B: "a-1.out" \/ "a-2.out"

-- C is the difference of the file b-1.out and the set B
C: "b-1.out" - B

-- D is the symmetric difference of two files (elements in one but not both)
D: "a-1.out" >< "a-2.out"

Files with these definitions are usually suffixed .setdown and fed to the executable:

setdown path/to/mydefinitions.setdown

Output

When setdown runs, it creates an output/ directory next to your .setdown file. Each named definition produces a result file in that directory named after the definition with a .txt extension — for example, a definition called Overlap produces output/Overlap.txt. The file contains one element per line, sorted and de-duplicated.

Intermediate results for sub-expressions are computed in a scratch output/processing/ directory, which is removed automatically once the run finishes. Pass --keep-processing to leave it in place for debugging, and --show-transient to have those intermediate results included in the summary table setdown prints at the end of a run.

You can choose a different output directory with --output, given relative to the .setdown file, not the current working directory:

setdown --output=results mydefinitions.setdown

Command-line flags

setdown evaluates a .setdown definitions file to perform set operations
(intersection, union, difference) on line-based text files, writing one
result file per definition to an output directory.

setdown [OPTIONS]

Common flags:
  -o --output[=DIR]               Directory in which to place output files,
                                  relative to your .setdown file. Defaults to
                                  'output' if omitted.
  -i --input=definitions.setdown  The .setdown definitions file to evaluate.
                                  If omitted, setdown looks for a single
                                  .setdown file in the current directory and
                                  uses it automatically. Exits with an error if
                                  zero or more than one are found.
     --show-transient             Also show intermediate results for
                                  sub-expressions generated internally to
                                  evaluate your definitions. Useful for
                                  debugging complex .setdown files.
     --keep-processing            Keep the processing/ subdirectory after the
                                  run completes instead of deleting it. Useful
                                  for inspecting intermediate files when
                                  debugging.
  -? --help                       Display help message
  -V --version                    Print version information

Troubleshooting

Setdown prints a short error message to stdout and exits with a non-zero code when something goes wrong:

All file paths in error messages are relative to the .setdown file, not the current working directory. See examples/parse-error and examples/cycle-detection for these in action.

Building the code

Setdown is a Haskell library (the set language, parser, and evaluation engine) plus a thin setdown executable, built with Stack:

stack build

Run the three test suites — unit, property-based, and golden — with:

stack test

The same build-and-test steps run in CI on every push and pull request; see .github/workflows.

To run setdown during development, without installing it:

stack exec -- setdown --help
stack exec -- setdown mydefinitions.setdown

Contributing

Contributions are welcome. The preferred workflow is:

1. Open an issue describing what you intend to fix or improve.
2. Write the code.
3. Open a pull request and ask Robert Massaioli to review it.
4. Iterate until the code is clean and merged.
5. Celebrate!

Design proposals and background for larger changes live under ai-planning/ if you'd like context before picking something up.