Archives and compression on Linux: tar, gzip, xz, zstd and zip

People coming from Windows are used to thinking of «zip» as a single operation. On Linux the job is split into two steps: first you archive, gathering many files into one while keeping the folder structure, permissions and dates; then you compress, reducing the size. The first job is done by tar, the second by one of several compressors. Understanding this split makes file names such as .tar.gz or .tar.zst clear, along with the options to use.

tar: the archiver

The name comes from tape archive: it was born to write data to tape. Today tar creates a file that contains other files, with all their metadata. The three fundamental operations each have a letter:

• c creates an archive;
• x extracts;
• t lists the contents without extracting.

To these you add f, followed by the archive file name, and v, which shows files as they are processed. For example:

tar -cvf project.tar project/
tar -tvf project.tar
tar -xvf project.tar

The file project.tar is not compressed: it takes up roughly as much space as the original files.

Adding compression

tar can pass the data straight to a compressor with one extra letter:

So, to create a gzip-compressed archive:

tar -czvf site-backup.tar.gz /var/www/site

When extracting, recent versions of GNU tar detect the compression type on their own, so tar -xf archive.tar.xz works without specifying J.

Which compressor to choose

Compressors make different trade-offs between speed and output size:

• gzip is the most widespread and compatible, fast at both compressing and decompressing, with a decent ratio. It is the safe choice when the archive is going to someone else or will be opened on any system.
• bzip2 compresses a little better than gzip but is slower; it is used less today.
• xz produces some of the smallest files, but compression is slow and needs a lot of memory at high levels. Many projects use it to distribute source code, because it is compressed once and downloaded many times.
• zstd offers very fast decompression and a wide range of levels, from extremely quick to very compact. Several distributions use it for their own packages. For frequent backups it is often the best compromise.

All of them accept a level: gzip -9 for maximum compression, zstd -19 for a very compact archive. You can also use more than one processor: zstd -T0 uses every available core, and xz -T0 does the same.

One thing worth knowing: compressing files that are already compressed, such as JPEG photos, videos, PDFs or ZIP archives, yields almost no gain. In those cases plain tar without compression is enough, and at least gathers everything into a single file.

Practical tar options

A few options I use often:

• -C folder extracts into a folder other than the current one: tar -xzf archive.tar.gz -C /tmp/test;
• --exclude skips files or folders: --exclude='*.log' --exclude='node_modules';
• giving one or more paths after the archive name extracts only those files: tar -xzf backup.tar.gz project/config.yml;
• -p preserves permissions on extraction, the default behaviour when running as root.

A detail about paths: GNU tar strips the leading / from absolute paths when creating an archive, and says so with a message. That way, on extraction, files land in the current folder rather than overwriting system files.

Checking before extracting

An archive received from someone else should be listed before it is extracted:

tar -tzf archive.tar.gz | head

There are two problems to avoid. The first is an archive with no top-level folder: extracted in your home folder, it scatters dozens of files among yours. The second, more serious, is paths containing ../, which would try to write outside the destination folder. Modern versions of tar block or flag them, but the habit of looking first is still the best defence. Extracting into an empty folder created for the purpose, with -C, solves both.

zip and 7z: for exchanging with other systems

To send files to people using Windows or macOS, ZIP remains the most convenient format, because both systems open it without extra software:

zip -r documents.zip documents/
unzip -l documents.zip
unzip documents.zip -d destination/

ZIP compresses each file separately and doesn't preserve Unix permissions well, so it isn't suitable for a system backup. The 7z format, handled by the 7z command, compresses very well and supports AES encryption of archives; it is useful when exchanging files with 7-Zip users.

Checking integrity

Before deleting the originals, check that the archive is readable: gzip -t archive.tar.gz, xz -t, zstd -t or unzip -t verify that the data decompresses without errors. For an important backup it is also worth extracting it into a test folder and comparing a few files.

In short

• tar gathers files with their metadata; the compressor reduces the size.
• c, x and t create, extract and list; z, J and --zstd add compression.
• gzip for compatibility, xz for the smallest files, zstd for speed and frequent backups.
• Someone else's archive is listed before extraction, ideally into an empty folder.
• ZIP is the format for exchanging files with Windows and macOS, not for system backups.