zstd – A Fast Data Compression Algorithm Used By Facebook

Zstandard (also known as zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by Facebook. It is a lossless compression algorithm written in C (there is a re-implementation in Java) – its thus a native Linux program.

Read Also: 10 7zip (Data Comperssion) Command Examples in Linux

When required, it can trade compression speed for stronger compression ratios (compression speed vs compression ratio trade-off can be configured by small increments), vice versa. It has a special mode for small data compression, known as dictionary compression, and can build dictionaries from any sample set provided. It comes with a command line utility for creating and decoding .zst, .gz, .xz and .lz4 files.

Importantly, Zstandard has a rich collection of APIs, supports almost all popular programming languages including Python, Java, JavaScript, Nodejs, Perl, Ruby, C#, Go, Rust, PHP, Switft, and lots more.

It is actively used to compress large volumes of data in multiple formats and use cases in Facebook; services such as Amazon Redshift data warehousing; databases such as Hadoop and Redis; the Tor network and many other applications including games.

The following results are obtained by doing several fast compression algorithms tests on a server running Linux Debian using lzbench, an open-source in-memory benchmark tool.

Zstandard Compression Testing
Zstandard Compression Testing

How to Install Zstandard Compression Tool in Linux

To install Zstandard on a Linux distribution, you need to compile it from sources, but before that first you need to install the necessary development tools on your system using your distribution package manager as shown.

$ sudo apt update && sudo apt install build-essential		#Ubuntu/Debian
# yum group install "Development Tools" 			#CentOS/REHL
# dnf groupinstall "C Development Tools and Libraries"		#Fedora 22+

Once all the needed development tools installed, now you can download the source package, move into the local repo directory, build the binary and install it as shown.

$ cd ~/Downloads
$ git clone https://github.com/facebook/zstd.git
$ cd zstd
$ make
$ sudo make install 

Once Zstandard installed, now we can move further to learn some basic usage of Zstd command examples in the following section.

Learn 10 Zstd Command Usage Examples in Linux

Zstd’s command line syntax is generally similar to that of gzip and xz tools, with a few differences.

1. To create a .zst compression file, simply provide a filename to compress it or use the -z flag also means compress, which is the default action.

$ zstd etcher-1.3.1-x86_64.AppImage 
OR
$ zstd -z etcher-1.3.1-x86_64.AppImage 

2. To decompress a .zst compression file, use the -d flag or the unzstd utility as shown.

$ zstd -d etcher-1.3.1-x86_64.AppImage.zst 
OR
$ unzstd etcher-1.3.1-x86_64.AppImage.zst 

3. To remove source file after an operation, by default, the source file is not deleted after successful compression or decompression, to delete it, use the --rm option.

$ ls etcher-1.3.1-x86_64.AppImage
$ zstd --rm  etcher-1.3.1-x86_64.AppImage
$ ls etcher-1.3.1-x86_64.AppImage

4. To set a compression level, zstd has a number of operation modifiers, for instance you can specify a compression level as -6(a number 1-19, default is 3) as shown.

$ zstd -6 --rm etcher-1.3.1-x86_64.AppImage

5. To set a compression speed, zstd has a compression speed ratio 1-10, the default compression speed is 1. You can trade compression ratio for compression speed with the --fast option, the higher the number the faster the compression speed.

$ zstd --fast=10 etcher-1.3.1-x86_64.AppImage

6. To display information about a compressed file, use the -l flag, which is used to display information about a compressed file, for example.

$ zstd -l etcher-1.3.1-x86_64.AppImage.zst

7. To test the integrity of a compressed files, use the -t flag as shown.

$ zstd -t etcher-1.3.1-x86_64.AppImage.zst

8. To enable verbose mode, use the -v option.

$ zstd -v -5 etcher-1.3.1-x86_64.AppImage

9. To use other file compression or decompression formats such as gzip, xz, lzma, and lz4, using the --format=FORMAT as shown.

$ zstd -v --format=gzip etcher-1.3.1-x86_64.AppImage
$ zstd -v --format=xz  etcher-1.3.1-x86_64.AppImage

10. To set a zstd process priority to real-time, you can use the option –priority=rt as shown.

$zstd --priority=rt etcher-1.3.1-x86_64.AppImage

The -r flag instructs zstd to operate recursively on dictionaries. You can find lots of useful and advanced options, how to read or create dictionaries by consulting the zstd man page.

$ man zstd

Zstandard Github Repository: https://github.com/facebook/zstd

Zstandard is a fast real-time, lossless data compression algorithm and compression tool which offers high compression ratios. Try it out and share your thoughts about it or ask questions via the feedback form below.

If you read this far, tweet to the author to show them you care. Tweet a thanks
Aaron Kili
Aaron Kili is a Linux and F.O.S.S enthusiast, an upcoming Linux SysAdmin, web developer, and currently a content creator for TecMint who loves working with computers and strongly believes in sharing knowledge.

Each tutorial at TecMint is created by a team of experienced Linux system administrators so that it meets our high-quality standards.

Join the TecMint Weekly Newsletter (More Than 156,129 Linux Enthusiasts Have Subscribed)
Was this article helpful? Please add a comment or buy me a coffee to show your appreciation.

2 thoughts on “zstd – A Fast Data Compression Algorithm Used By Facebook”

  1. I think there is a mistake in point 2

    It says, To decompress a .zst compression file, use the -d flag or the unzstd utility as shown.

    $ zstd -d etcher-1.3.1-x86_64.AppImage 
    OR
    $ unzstd etcher-1.3.1-x86_64.AppImage 
    

    Instead it should be

    To decompress a .zst compression file, use the -d flag or the unzstd utility as shown.

    $ zstd -d etcher-1.3.1-x86_64.AppImage.zst
    OR
    $ unzstd etcher-1.3.1-x86_64.AppImage.zst
    
    Reply

Got something to say? Join the discussion.

Thank you for taking the time to share your thoughts with us. We appreciate your decision to leave a comment and value your contribution to the discussion. It's important to note that we moderate all comments in accordance with our comment policy to ensure a respectful and constructive conversation.

Rest assured that your email address will remain private and will not be published or shared with anyone. We prioritize the privacy and security of our users.