Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How did I miss zstd?

Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.

    gzip  7.3%  21s  9s
    bzip2 4.6% 251s 50s
    bzip3 3.3%  82s 69s
    zstd  6.9%   2s  3s
    lzma  4.7%  51s  3s
 help



At what levels? There’s no guarantee that the default compression level is comparable. You have to normalize by time spent compressing.

reply


But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.

reply


[flagged]


No it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).

reply


zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)

reply


zstd with better compression level would be nice - these numbers are not really comparable since both time and compression level are too different

reply




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: