Archiving and compression: Difference between revisions

From ArchWiki
(→‎Compression only: expansion flag: add zlib)
(→‎See also: update list of application link)
(25 intermediate revisions by 2 users not shown)
Line 8: Line 8:
These tools are often used in sequence by firstly creating an archive file and then compressing it.
These tools are often used in sequence by firstly creating an archive file and then compressing it.


Of course there are also [[#Archiving and compression|programs that do both]], which additionally tend to offer encryption, error detection and recovery.
Of course there are also [[#Archiving and compression|tools that do both]], which tend to additionally offer encryption, error detection and recovery.


== Archiving only ==
== Archiving only ==
{{Expansion|Mention {{Pkg|libarchive}}.}}


{| class=wikitable
{| class=wikitable
! Name !! Packages !! Manual !! Description
! Name !! Packages !! Manuals !! Description
|-
| [[Wikipedia:tar (computing)|tar]] || {{Pkg|coreutils}} || {{man|1|tar}} || GNU utility for manipulating the ubiquitous tar archives (tarballs), see [[tar]] for usage examples.
|-
|-
| [[Wikipedia:ar (Unix)|ar]] || {{Pkg|binutils}} || {{man|1|ar}}|| Legacy Unix archiver before ''tar''. Today only used for creating [[Wikipedia:Static library|static library]] files.
| [[Wikipedia:ar (Unix)|ar]] || {{Pkg|binutils}} || {{man|1|ar}}|| Legacy Unix archiver before ''tar''. Today only used for creating [[Wikipedia:Static library|static library]] files.
|-
|-
| [[Wikipedia:cpio|cpio]] || {{Pkg|cpio}} || {{man|1|cpio}} || File archiver via stdin/stdout, supports cpio and tar formats.
| [[Wikipedia:cpio|cpio]] || {{Pkg|cpio}} || {{man|1|cpio}} || File archiver via stdin/stdout, supports cpio and tar formats.
|-
| [http://dar.linux.free.fr/ DAR] || {{AUR|dar}} || {{man|1|dar|url=http://dar.linux.free.fr/doc/man/dar.html}} || Archiver to backup large live filesystems, takes care of hard links, [[extended attributes]], sparse files and inode types.
|-
| GNU [[Wikipedia:tar (computing)|tar]] || {{Pkg|coreutils}} || {{man|1|tar}} || GNU utility for manipulating the ubiquitous tar archives (tarballs), see [[tar]] for usage examples.
|-
| [http://libarchive.org/ libarchive] || {{Pkg|libarchive}} || {{man|1|bsdtar}}<br>{{man|1|bsdcpio}} || Implementation of ''tar'' and ''cpio'' that also offers a library. Used by [[pacman]] and [[mkinitcpio]].
|}
|}


{{Note|When GNU '''tar''' is given a compressed file it automatically invokes the right compression program based on the file extension. bzip2, xz, lzip, lxma, lzop, gzip and compress are supported.}}
{{Tip|Both GNU and BSD tar automatically do decompression delegation for bzip2, compress, gzip, lzip, lzma and xz compressed archives. When creating archives both support the {{ic|-a}} switch to automatically filter the created archive through the right compression program based on the file extension. While BSD tar recognizes compression formats based on the format, GNU tar only guesses based on the file extension.}}


== Compression tools ==
== Compression tools ==


=== Compression only ===
=== Compression only ===
{{Expansion|Add {{Pkg|zlib}}.}}


These compression programs implement their own file format.
These compression programs implement their own file format.
Line 41: Line 41:
| [[Wikipedia:gzip|gzip]] || {{Pkg|gzip}} || {{man|1|gzip}} || .gz, .z|| .tgz, .taz|| GNU zip, based on [[Wikipedia:DEFLATE|DEFLATE]] algorithm.|| {{Pkg|pigz}}  
| [[Wikipedia:gzip|gzip]] || {{Pkg|gzip}} || {{man|1|gzip}} || .gz, .z|| .tgz, .taz|| GNU zip, based on [[Wikipedia:DEFLATE|DEFLATE]] algorithm.|| {{Pkg|pigz}}  
|-
|-
| [[lrzip]] || {{Pkg|lrzip}} || {{man|1|lrzip}} || .lrz || || Improved version of [[Wikipedia:rzip|rzip]], uses multiple compression algorithms. ||  
| [[lrzip]] || {{Pkg|lrzip}} || {{man|1|lrzip}} || .lrz || || Improved version of [[Wikipedia:rzip|rzip]], uses multiple algorithms. || is multithreaded
|-
|-
| [[Wikipedia:lzip|lzip]] || {{Pkg|lzip}} || {{man|1|lzip}} || .lz || || Uses [[Wikipedia:LZMA|LZMA]]. ||
| [[Wikipedia:LZ4|LZ4]] || {{Pkg|lz4}} || {{man|1|lz4}} || .lz4 || || Written in C, focused on compression and decompression speed. || is multithreaded
|-
|-
| [[Wikipedia:lzop|lzop]] || {{Pkg|lzop}} || {{man|1|lzop}} || .lzop || .tzo || Uses [[Wikipedia:Lempel–Ziv–Oberhumer|LZO]].||
| [[Wikipedia:lzip|lzip]] || {{Pkg|lzip}} || {{man|1|lzip}} || .lz ||  || Uses [[Wikipedia:LZMA|LZMA]]. || {{AUR|plzip}}
|-
| [[Wikipedia:lzop|lzop]] || {{Pkg|lzop}} || {{man|1|lzop}} || .lzop || .tzo || Uses the [[Wikipedia:Lempel–Ziv–Oberhumer|LZO]] library ({{Pkg|lzo}}).||
|-
|-
| [[Wikipedia:xz|xz]] || {{Pkg|xz}} || {{man|1|xz}} || .xz, .lzma || .txz, .tlz|| Uses [[Wikipedia:LZMA|LZMA]], default for GNU {{Pkg|coreutils}} and kernel archive files.|| {{Pkg|pixz}},  {{aur|pxz}}  
| [[Wikipedia:xz|xz]] || {{Pkg|xz}} || {{man|1|xz}} || .xz, .lzma || .txz, .tlz|| Uses [[Wikipedia:LZMA|LZMA]], default for GNU {{Pkg|coreutils}} and kernel archive files.|| {{Pkg|pixz}},  {{aur|pxz}}  
Line 59: Line 61:
! Name !! Packages !! Manual !! Ext !!  Description
! Name !! Packages !! Manual !! Ext !!  Description
|-
|-
| [[Wikipedia:7z|7z]] || {{Pkg|p7zip}} || {{man|1|7z}} || .7z || Popular OpenSource tool on non-Linux machines. See [[p7zip]].
| [[Wikipedia:7z|7z]] || {{Pkg|p7zip}} || {{man|1|7z}} || .7z || POSIX port of 7-zip's command-line. See [[p7zip]].
|-
|-
| [[Wikipedia:RAR (file format)|RAR]] || {{AUR|rar}}, {{Pkg|unrar}} || {{man|1|rar|url=}} || .rar || Popular Windows tool available for Linux. See [[RAR]].
| [[Wikipedia:RAR (file format)|RAR]] || {{AUR|rar}}, {{Pkg|unrar}} || {{man|1|rar|url=}} || .rar || Both the format and the [[rar]] utility are proprietary.
|-
|-
| [[Wikipedia:Zip (file format)|ZIP]] || {{Pkg|zip}}, {{Pkg|unzip}} || {{man|1|zip}}, {{man|1|unzip}} || .zip || Widely used outside of the Linux-world.
| [[Wikipedia:Zip (file format)|ZIP]] || {{Pkg|zip}}, {{Pkg|unzip}} || {{man|1|zip}}, {{man|1|unzip}} || .zip || Widely used outside of the Linux world.
|-
|-
| [https://theunarchiver.com/ Unarchiver] || {{Pkg|unarchiver}} || {{man|1|unar}}, {{man|1|lsar}} || ''many'' || Command-line tool of a Mac application, supports over 40 archive formats.
| [https://theunarchiver.com/ Unarchiver] || {{Pkg|unarchiver}} || {{man|1|unar}}, {{man|1|lsar}} || ''many'' || Command-line tool of a Mac application, supports over 40 archive formats.
|}
=== Esoteric, rare or deprecated tools ===
{| class=wikitable
! Name !! Packages !! Ext !! Description
|-
| [[Wikipedia:ARC (file format)|ARC]] || {{AUR|arc}} || .arc, .ark|| Was very popular during the early days of the dial-up BBS. Superseded by ZIP.
|-
| [[Wikipedia:ARJ|ARJ]] || {{Pkg|arj}} || .arj|| An archiver used on DOS/Windows in mid-1990s. This is an open source clone.
|-
| [[Wikipedia:compress|compress]] || {{AUR|ncompress}} || .Z || The classic unix compression utility which can handle the ancient .Z archive.
|-
| [[Wikipedia:Parchive|PAR2]] || {{Pkg|par2cmdline}} || .par2|| Parity archiver for increased data integrity. See also [[Parchive]].
|-
|-
| [[Wikipedia:ZPAQ|ZPAQ]] || {{AUR|zpaq}} || {{man|1|zpaq|url=http://mattmahoney.net/dc/zpaqdoc.html}} || .zpaq || A high compression ratio archiver written in C++, uses several algorithms.
|}
|}


Line 149: Line 138:


Otherwise you can use the {{Pkg|file}} tool, see {{man|1|file}}.
Otherwise you can use the {{Pkg|file}} tool, see {{man|1|file}}.
== Esoteric, rare or deprecated tools ==
{| class=wikitable
! Name !! Packages !! Ext !! Description
|-
| [[Wikipedia:ARC (file format)|ARC]] || {{AUR|arc}} || .arc, .ark|| Was very popular during the early days of the dial-up BBS. Superseded by ZIP.
|-
| [[Wikipedia:ARJ|ARJ]] || {{Pkg|arj}} || .arj|| An archiver used on DOS/Windows in mid-1990s. This is an open source clone.
|-
| [[Wikipedia:compress|compress]] || {{AUR|ncompress}} || .Z || The classic unix compression utility which can handle the ancient .Z archive.
|-
| [[Wikipedia:LHA (file format)|LHA]] || {{AUR|lha}} || .lzh, .lha || Format popular in Japan, archiver to create LH-7 format archives. 32-bit only (requires [[multilib]]).
|-
| [[Wikipedia:Parchive|PAR2]] || {{Pkg|par2cmdline}} || .par2|| Parity archiver for increased data integrity. See also [[Parchive]].
|-
| [[Wikipedia:shar|shar]] || {{Pkg|sharutils}} || .shar || Creates self-extracting archives that are valid shell scripts.
|-
| [[Wikipedia:Zoo_(file_format)|Zoo]] || {{AUR|zoo}} || .zoo || Was mostly popular on the [[Wikipedia:OpenVMS|OpenVMS]] operating system before PKZIP became popular.
|}
== Compression libraries ==
* {{App|[[Wikipedia:zlib|zlib]]|Compression library implementing the deflate compression method found in gzip and PKZIP.|https://www.zlib.net/|{{Pkg|zlib}}}}
* {{App|[[Wikipedia:Zopfli|Zopfli]]|High compress ratio file compressor from Google, using a deflate-compatible algorithm called zopfli.|https://github.com/google/zopfli|{{aur|zopfli-git}}}}


== See also ==
== See also ==


* [[List of applications/Utilities#Archiving and compression tools]]
* [[List of applications/Utilities#Archive managers]]
* [[List of applications/Multimedia#Image compression]]
* [[Wikipedia:Comparison of file archivers]]
* [[Wikipedia:Comparison of file archivers]]
* [[Wikipedia:List of archive formats]]
* [[Wikipedia:List of archive formats]]
* [[Wikipedia:Comparison of archive formats]]
* [[Wikipedia:Comparison of archive formats]]

Revision as of 09:43, 29 July 2018

The traditional Unix archiving and compression tools are separated according to the Unix philosophy:

  • A file archiver combines several files into one archive file, e.g. tar.
  • A compression tool compresses and decompresses data.

These tools are often used in sequence by firstly creating an archive file and then compressing it.

Of course there are also tools that do both, which tend to additionally offer encryption, error detection and recovery.

Archiving only

Name Packages Manuals Description
ar binutils ar(1) Legacy Unix archiver before tar. Today only used for creating static library files.
cpio cpio cpio(1) File archiver via stdin/stdout, supports cpio and tar formats.
DAR darAUR dar(1) Archiver to backup large live filesystems, takes care of hard links, extended attributes, sparse files and inode types.
GNU tar coreutils tar(1) GNU utility for manipulating the ubiquitous tar archives (tarballs), see tar for usage examples.
libarchive libarchive bsdtar(1)
bsdcpio(1)
Implementation of tar and cpio that also offers a library. Used by pacman and mkinitcpio.
Tip: Both GNU and BSD tar automatically do decompression delegation for bzip2, compress, gzip, lzip, lzma and xz compressed archives. When creating archives both support the -a switch to automatically filter the created archive through the right compression program based on the file extension. While BSD tar recognizes compression formats based on the format, GNU tar only guesses based on the file extension.

Compression tools

Compression only

These compression programs implement their own file format.

Name Packages Manual Ext Tar ext Description Parallel implementations
bzip2 bzip2 bzip2(1) .bz2, .bz .tbz2, .tbz Uses the Burrows–Wheeler algorithm. lbzip2, pbzip2
gzip gzip gzip(1) .gz, .z .tgz, .taz GNU zip, based on DEFLATE algorithm. pigz
lrzip lrzip lrzip(1) .lrz Improved version of rzip, uses multiple algorithms. is multithreaded
LZ4 lz4 lz4(1) .lz4 Written in C, focused on compression and decompression speed. is multithreaded
lzip lzip lzip(1) .lz Uses LZMA. plzipAUR
lzop lzop lzop(1) .lzop .tzo Uses the LZO library (lzo).
xz xz xz(1) .xz, .lzma .txz, .tlz Uses LZMA, default for GNU coreutils and kernel archive files. pixz, pxzAUR
  • Parallel implementations offer improved speeds by using multiple CPU cores.
  • Tar extensions refers to compressed archives where tar and the compression tool is used, e.g. .tzo is .tar.lzo.

Archiving and compression

Name Packages Manual Ext Description
7z p7zip 7z(1) .7z POSIX port of 7-zip's command-line. See p7zip.
RAR rarAUR, unrar rar(1) .rar Both the format and the rar utility are proprietary.
ZIP zip, unzip zip(1), unzip(1) .zip Widely used outside of the Linux world.
Unarchiver unarchiver unar(1), lsar(1) many Command-line tool of a Mac application, supports over 40 archive formats.
ZPAQ zpaqAUR zpaq(1) .zpaq A high compression ratio archiver written in C++, uses several algorithms.

Feature charts

Decompress

Name gzip bzip2 ZIP compress pack CAB ARJ
gzip Yes No Yes Yes Yes No No
p7zip Yes Yes Yes No Yes Yes Yes
unarchiver Yes Yes Yes Yes No Yes partial

Usage comparison

Archiving only

Name Create archive Extract archive List content
tar(1) tar cfv archive.tar file1 file2 tar xfv archive.tar tar -tvf archive.tar
cpio(1) ls file1 file2 | cpio -o > archive.cpio cpio -i -vd < archive.cpio cpio -t < archive.cpio

Compression only

Name Compress Decompress Decompress to stdout
bzip2(1) bzip2 file bzip2 -d file.bz2 bzcat file.bz2
gzip(1) gzip file gzip -d file.gz zcat file.gz
lrzip(1) lrzip file
lrztar folder
lrzip -d file.lrz
lrztar -d folder.tar.lrz
lrzcat file.lrz
xz(1) xz file xz -d file.xz xzcat file.xz

Archiving and compression

Name Compress Decompress Decompress to stdout List content
7z(1) 7z a archive.7z file1 file2 7z x archive.7z 7z e -so archive.7z file1 7z l archive.7z
rar(1) & unrar rar a archive.rar file1 file2 rar x archive.rar rar p -inul archive.rar file1 rar l archive.rar
zip(1), unzip(1) zip archive.zip file1 file2 unzip archive.zip unzip -p archive.zip file1 unzip -l archive.zip

Convenience tools

  • atool — Script for managing file archives of various types.
https://www.nongnu.org/atool/ || atool
  • dtrx — An intelligent archive extraction tool.
https://brettcsmith.org/2007/dtrx/ || dtrxAUR
  • unp — Command line tool that can unpack archives easily.
https://github.com/mitsuhiko/unp || python-unpAUR
  • unpack — Wrapper script for handling multiple archive formats.
https://github.com/githaff/unpack || unpack-gitAUR

Determining archive format

To extract an archive, its file format needs to be determined. If the file is properly named you can deduce its format from the file extension.

Otherwise you can use the file tool, see file(1).

Esoteric, rare or deprecated tools

Name Packages Ext Description
ARC arcAUR .arc, .ark Was very popular during the early days of the dial-up BBS. Superseded by ZIP.
ARJ arj .arj An archiver used on DOS/Windows in mid-1990s. This is an open source clone.
compress ncompressAUR .Z The classic unix compression utility which can handle the ancient .Z archive.
LHA lhaAUR .lzh, .lha Format popular in Japan, archiver to create LH-7 format archives. 32-bit only (requires multilib).
PAR2 par2cmdline .par2 Parity archiver for increased data integrity. See also Parchive.
shar sharutils .shar Creates self-extracting archives that are valid shell scripts.
Zoo zooAUR .zoo Was mostly popular on the OpenVMS operating system before PKZIP became popular.

Compression libraries

  • zlib — Compression library implementing the deflate compression method found in gzip and PKZIP.
https://www.zlib.net/ || zlib
  • Zopfli — High compress ratio file compressor from Google, using a deflate-compatible algorithm called zopfli.
https://github.com/google/zopfli || zopfli-gitAUR

See also