Talk:Archiving and compression

TODO: New parallel gzip decompression tool: pragzip/igzip

As on 2022, pigz(1) still does not support true parallel decompression of gzip stream. A recently implemented decompression-only Python/C++ gzip library (incl. cli utility in C++ called igzip) that is capable to decompress pigz/bgzip(1)'s output parallelly.

Impl details: https://github.com/madler/pigz/issues/36#issuecomment-1004248819

The implementation is nearly stable but no distro is packaging it per repology (the repository layout is a bit confusing too, shared by both ibzip and igzip and two library.).

As a pending backend of the already pkg'd ratarmount^AUR, once it get pkg'd, should it be added to the table as this increased the performance on decompression of large gzip file significantly.

-- Jlhs (talk) 15:12, 13 September 2022 (UTC)

Thank you for linking to my comment about indexed_bzip2 / rapidgzip on the pigz issue. Some short corrections: The Python/C++ utility for parallel decompression is called rapidgzip (at the point of the comment it was called pragzip). It also got its own repository by now even though it has still the same structure as indexed_bzip2. On the other hand, igzip is the command line utility of the ISA-L library, which is also a quite fast gzip decompressor, twice as fast as pigz and roughly 4x as fast as gzip on my system, and roughly as fast as libdeflate.

I think that rapidgzip is at a point at which I should spent some time trying to get it into repositories. I'll probably try a bit with AUR and then maybe even try with Debian. I never added a package to either, though.

--Mxmlnkn (talk) 19:51, 8 September 2023 (UTC)