Hey folks, thought I'd share this little bit of info I discovered. For the longest time I've been using pigz (a gzip program which takes advantage of multicore capability) to compress my database backups. Out of curiosity (and that my backups keep getting larger) I wanted to see if using the xz format was possible. The xz program provided by xz-utils on my system isn't multithreaded (from the man page):
-T threads, --threads=threads Specify the number of worker threads to use. The actual number of threads can be less than threads if using more threads would exceed the memory usage limit. Multithreaded compression and decompression are not implemented yet, so this option has no effect for now. As of writing (2010-09-27), it hasn't been decided if threads will be used by default on multicore systems once support for threading has been implemented. Comments are welcome. The com- plicating factor is that using many threads will increase the memory usage dramatically. Note that if multithreading will be the default, it will probably be done so that single-threaded and multithreaded modes produce the same output, so compression ratio won't be significantly affected if threading will be enabled by default. This stinks because the server running the backups has 8 physical cores and no shortage of memory. I found out that 7z and 7za from p7zip create xz archives using multiple threads by default. I did some experiments on a dual-core system (Intel Core2 Duo E6420) with various compression programs and levels using a 2908748kb (2.7GB) database dump from my moodle server. The 7z command-line options are a little weird, but workable. Let me break down one of the commands I used: 7za a dummy -txz -mx3 -si -so <db.sql >7z3_db.sql.xz
| Command Segment | Explanation |
|---|---|
| 7za | 7za is the 7-zip binary. The difference between 7z and 7za is 7za doesn't load any plugins (only uses native algorithms). For our purposes there really is no difference. |
| a | Add files to archive |
| dummy | Necessary when compressing to stdout with the -so option. |
| -txz | Sets the type of archive to xz. |
| -mx3 | Sets the compression level to 3 (valid range is 0-9). |
| -si | Compress data from stdin |
| -so | Compress data to stdout |
| <db.sql | Read db.sql into stdin (I know 7z can read files directly, but when I'm dumping the SQL it's going into stdin and I wanted my test case to be from stdin too). |
| >7z3_db.sql.xz | Write stdout to 7z3_db.sql.xz. In my backup script, this will be piped over SSH. |
Below are the results.
| Command | Run Time (s) | Size (kb) | Size (mb) | Ratio |
|---|---|---|---|---|
| pigz -1 <db.sql >pigz1_db.sql.gz | 51.83 | 543796 | 531.05 | 18.70% |
| lzop <db.sql >lzop_db.sql.lzop | 53.82 | 755204 | 737.50 | 25.96% |
| gzip -1 <db.sql >gzip1_db.sql.gz | 63.75 | 545084 | 532.31 | 18.74% |
| pigz <db.sql >pigz_db.sql.gz | 81.89 | 417484 | 407.70 | 14.35% |
| 7za a dummy -txz -mx1 -si -so <db.sql >7z1_db.sql.xz | 117.08 | 335172 | 327.32 | 11.52% |
| 7za a dummy -txz -mx2 -si -so <db.sql >7z2_db.sql.xz | 124.18 | 326076 | 318.43 | 11.21% |
| 7za a dummy -txz -mx3 -si -so <db.sql >7z3_db.sql.xz | 150.44 | 320780 | 313.26 | 11.03% |
| 7za a dummy -txz -mx4 -si -so <db.sql >7z4_db.sql.xz | 237.03 | 317316 | 309.88 | 10.91% |
| gzip <db.sql >gzip_db.sql.gz | 266.46 | 418176 | 408.38 | 14.38% |
| pbzip2 -1 <db.sql >pbzip21_db.sql.bz2 | 286.09 | 342164 | 334.14 | 11.76% |
| pigz --best <db.sql >pigzbest_db.sql.gz | 293.68 | 407296 | 397.75 | 14.00% |
| pbzip2 -9 <db.sql >pbzip29_db.sql.bz2 | 391.75 | 302644 | 295.55 | 10.40% |
| bzip2 -1 <db.sql >bzip1_db.sql.bz2 | 492.68 | 341880 | 333.87 | 11.75% |
| gzip -9 <db.sql >gzip9_db.sql.gz | 600.57 | 408304 | 398.73 | 14.04% |
| bzip2 -9 <db.sql >bzip9_db.sql.bz2 | 632.29 | 302208 | 295.13 | 10.39% |
| xz -3 <db.sql >xz_db.sql.xz | 645.39 | 302028 | 294.95 | 10.38% |
We can see the runtimes of the single-threaded programs (with the exception of lzop) is by far the worst even on this dual-core system. lzop is almost twice as fast as the default pigz compression (-7) but results in a file almost twice as large. For 7za, the -mx1 compression level seems to be a fair candidate for replacement of pigz. It's only a little slower and in my case shaved off almost 3% of the file size. In my environment, I went with -mx3, because it saves almost 100mb per backup and delivers that in a reasonable time frame (for me).
TL;DR
Using 7za to create xz archives is a lot faster than xz and gzip (but not faster than pigz) on multicore systems and results in a much better compression ratio. Hope this helps someone. 😊
Update 2013-12-11T19:47+0000
- Added results for
pbzip2,bzip2,gzip, andpigz.
[link][14 comments]