Glyd

Lossless AI compression: 33% less GPU memory, bit for bit. Rust, C ABI, CLI; v0.26.0, October 2026. Every number below is measured on public data and reproducible from the repository.

Fastest or slowest? The table everyone uses

The 8.7 GB real-data corpus on AWS Graviton3, ratio · compress MB/s · decompress MB/s. One core, the way zstd's README reports it, then eight cores (Glyd's output decodes in parallel; a zstd or LZ4 frame decodes on one thread).

Reads: Glyd is the fastest, 3–7× zstd on a server. Writes: Glyd is slower, 0.77× zstd -3 (record mode 0.33×); LZ4 is the write-speed king on one core. Size: Glyd is the smallest wherever the data has structure, and ties elsewhere.

Against zstd, xz and brotli on 24 kinds of data

Every codec's own CLI on one thread, every decode compared byte for byte with its input. Green: Glyd's file is smaller.

Glyd wins where the data has structure: each field of a record becomes a column, and the deflate, snappy, zstd or JPEG inside a container is opened and re-created bit for bit, Parquet pages included. On plain text and executables it is zstd-class: within 2% at the fast tier, 1–15% larger than xz and brotli -11 at their strongest. Every codec's speed, the strong-tier and ratio-against-speed charts, and lz4, bzip2, zpaq and JPEG XL.

The store: compression across objects, not just inside them

Inside one object every codec sits on the same floor. The redundancy of object storage is between objects — builds, snapshots, dumps and releases that are near-copies of earlier ones. glyd-store bucket/ --put ... keeps each object as a delta against the stored object it most resembles when that pays; a read is at most five decodes. Measured on a 39 GB bucket, each object arriving in order, every one read back and compared:

Put runs at 620 MB/s end to end on ten cores; delete, compact and verify are there. A petabyte of such data in S3 Standard: $43K a year with zstd -3, $9.4K with the store. Chunk-level dedup, the backup approach, gains 1–4× on the same pairs.

Base mode: a version stored for the change, not the whole

Most stored bytes are versions — nightly dumps, snapshots, images, source trees. glyd --base old new parses every 32 MB of the new version with the old one's matching region as history and writes a stream that decodes with the same base. Measured on consecutive versions of real objects against zstd 1.5.7's own --patch-from, same machine, every rebuild byte-exact:

Compressed alone those versions are 33, 287 and 200 MB: a version costs 1–5% of what it did. Against zstd's fast patch Glyd stores 1.1–2.1× less at 1.8–3× the speed; --ultra --base stores 5–21% less than zstd -19's patch on every pair, at the plain --ultra speed. Content is found wherever it moved in the old version (a coarse map of the base picks each unit's region). Chunk-level dedup, the backup approach, gains only 1–4× on the same pairs.

A step costs 1.8 MB, 0.12% of the tree; the delta against a base 14 releases old is 3.6 MB, so a series needs no rebasing. A terabyte of such trees in S3 Standard: $37 a year compressed one by one with --max ($40 with zstd -3), $2.8–3.0 with base mode ($3.2–3.3 with zstd's patch).

Model weights and checkpoints

A safetensors file is opened tensor by tensor; against a base (--base, and the store, which finds a checkpoint's predecessor by its tensors), a checkpoint is stored for what changed. The Ryzen box, all cores, every file decoded and compared (report):

The smallest size a coder that sees each tensor on its own can reach is 682–684 MB for a Pythia checkpoint and 650 MB for Qwen2.5: Glyd is within 2–3% of it.

Training checkpoints with the optimizer's state (torch.save; Qwen2.5-0.5B with AdamW, 5.93 GB each) go in at 83% of their size alone and 77% against the checkpoint before; zstd -19 stores 92%, and its --patch-from stops at 2 GB.

The weights stay compressed in GPU memory and are rebuilt bit for bit as the model runs (gpu/): about a third of a bf16 model's memory back, and generation faster than bf16; perplexity as bf16's (17.0052 against 17.0015 on Wikipedia text).

Record mode: the data's own structure, not just its bytes

Byte-level matching (zstd, LZ4, Glyd's own levels) is a plateau: on real data zstd -19, xz and Glyd --ultra land within a few percent of each other. The redundancy of a log, a dump or a telemetry export sits in the same field of every record. glyd -r detects delimited lines, SQL dumps and JSON lines, turns them into one typed stream per field (integer, decimal and date-time deltas, dictionaries with recency ranks, text), compresses those, and rebuilds the bytes exactly. Logs whose lines vary in shape take a template: the line's text with a hole where every token holding a digit was, and the tokens as typed columns keyed by template and slot. Anything else is left as it is.

2.5–3.5× fewer bytes than the fast tier (LZ4, zstd -3), 1.5–2× fewer than zstd -19, written at 340–690 MB/s on 10 cores.

1.4–3.3× fewer bytes than zstd -3 and 1.1–2.1× fewer than zstd -19 with --ultra -r; --max -r writes at 260–460 MB/s and reads back at 1,200–1,400 MB/s. Sources and the script that fetches them: scripts/download_ext_corpus.sh.

The cold level: below the floor every LZ codec shares

On real data zstd -19, xz -9 and Glyd --ultra land within 5% of each other: that is the floor of byte matching. Context mixing — every bit predicted from many contexts at once and coded at the mixed probability, no parse — goes below it at 30–100× the CPU. glyd --cold is that level: eleven predictors with paq-style bit histories, two mixers, two SSE stages, 32 MB units coded in parallel at 1.2–1.3 MB/s per core each way. Measured on 64 MB slices against the strongest tools, every decode byte-checked:

The zpaq -m5 class within 3% either way, at 3–4× its speed per core; reads cost what writes cost. A terabyte is about 210 core-hours each way ($8 on Graviton3); 1.5–2× fewer bytes than zstd -19 saves $3–5 a year per raw terabyte in S3 Standard-IA, so the level pays for data kept two years or more and read a few times at most, or moved more than it is read.

The 8.7 GB corpus on AWS (Graviton3 and Sapphire Rapids, 8 threads)

--max -r stores 18% less than zstd -3 and 1% less than zstd -19 while writing 50× faster than zstd -19; Glyd's output decodes in parallel, so with 8 cores it reads 3–7× faster than a zstd frame. JSON events are not record-shaped (their bytes are hashes, ids and free text); the 128 MB long-distance matcher (--max --long, --ultra, the store) is what wins there.

A terabyte in S3 for a year

Compressed once by the CLI, read back once a month in the region, S3 Standard at list price, instance CPU seconds at the on-demand price (Graviton3):

Where zstd still wins, plainly: a hundred CPU-billed reads a month of record-mode data (its reads spend 2× zstd's CPU rebuilding the columns), single-core compression speed on match-dense data (--max at 0.89× zstd -3 on logs and table dumps on Graviton3; 1.01–1.05× on the rest, v0.14.5), small objects with dictionaries (1–3% larger, 1.4–2× slower per object), and data made of hashes.

How the numbers are made

• Correctness first: 106 tests, a million-mutation fuzz, files from every earlier release decoded unchanged, and scripts/verify_roundtrip.sh through the CLI on both machines with corrupted copies rejected or decoded exactly; base-mode streams refuse any other base.
• Fair comparison: zstd -3, zstd -19 and LZ4 on the same machines and thread counts, each on its fastest API, buffers kept; compressed bytes include every codec's own framing.
• A real workflow: compress, upload to S3, download, decompress, sha256 every file, at list prices.
• The floors are stated: JSON API events are 20% hashes, 10% random ids and 30% free text once compressed; Parquet is already zstd inside. Neither moves. Where zstd wins — write speed, small objects per second, reads at very high read rates — is written down too.

Full report with every table · Design notes · Changelog

The codec is BSD-3-Clause or GPL-2.0 at your option (the same licenses as zstd); the store (glyd-store) is under the Business Source License 1.1. Contact: suryakoritala1324@gmail.com.