shamochu: “Shuffle and merge overlapping chunks” lossless compression
Shamochu is short for “Shuffle and merge overlapping chunks” lossless compression.
The idea for the compression is:
Split the input array in chunks of a given size.
Rearrange the order of the chunks in order to optimize consecutive chunks overlaps.
Create a data table with the reordered chunks and an index table that maps the original chunk index to its offset in the data table.
Then the data can be accessed in \(\mathcal{O}(1)\) via a few bitwise operations and indexing two arrays.
The same operation can then be applied to the index table and may lead to further compression.
Trivial example (chunk size: 4):
[1, 2, 3, 4, 2, 3, 4, 5, 0, 1, 2, 3] # source data -> [[1, 2, 3, 4], [2, 3, 4, 5], [0, 1, 2, 3]] # make chunks -> [[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]] # rearrange to have best overlaps -> {data: [0, 1, 2, 3, 4, 5], offsets: [1, 2, 0]} # overlap chunks & compute # their offsets
Then we can retrieve the data from the original array at index i with the following formula:
mask = (1 << chunk_size) - 1 original[i] = data[offsets[i >> chunk_size] + (i & mask)]
Since the index array is itself quite repetitive with the real data, we can apply the compression a second time to the offsets table.
The complete algorithm optimizes the chunk sizes for both arrays in order to
get the lowest total data size. Given the chunks sizes cs_data
and
cs_offsets
:
We compute the corresponding masks:
mask_data = (1 << cs_data) - 1
andmask_offsets = (1 << cs_offsets) - 1
.
We can retrieve the original value at index
k
with the following formula:
data[ offsets1[ offsets2[i >> (cs_data + cs_offsets)] + ((i >> cs_data) & mask_offsets) ] + (i & mask_data) ];
Notes
This work took inspiration from “Fast character case conversion… or how to really compress sparse arrays” by Alexander Pankratov.
Downloads
- shamochu-0.1.0.0.tar.gz [browse] (Cabal source package)
- Package description (as included in the package)
Maintainer's Corner
For package maintainers and hackage trustees
Candidates
- No Candidates
Versions [RSS] | 0.1.0.0 |
---|---|
Change log | CHANGELOG.md |
Dependencies | base (>=4.14 && <5), containers (>=0.6.5 && <0.8), vector (>=0.13.0 && <0.14) [details] |
License | BSD-3-Clause |
Copyright | 2024 Pierre Le Marre |
Author | Pierre Le Marre |
Maintainer | dev@wismill.eu |
Category | Codec |
Home page | https://gitlab.com/Wismill/shamochu |
Uploaded | by wismill at 2024-06-14T10:06:40Z |
Distributions | NixOS:0.1.0.0 |
Downloads | 30 total (4 in the last 30 days) |
Rating | (no votes yet) [estimated by Bayesian average] |
Your Rating | |
Status | Docs available [build log] Last success reported on 2024-06-14 [all 1 reports] |