All posts by Mike Massonnet

Benchmarking Compression Tools

Comparison of several compression tools: lzop, gzip, bzip2, 7zip, and xz.
  • Lzop: small and very fast yet good compression.
  • Gzip: fast and good compression.
  • Bzip2: slow for both compression and decompression although very good compression.
  • 7-Zip: LZMA algorithm, slower than Bzip2 for compression but very good compression.
  • Xz: LZMA2, evolution of LZMA algorithm.

Preparation

  • Be skeptic about compression tools and wanna promote the compression tool
  • Compare quickly old and new compression tools and find interesting results
So much for the spirit, what you really need is to write a script (Bash, Ruby, Perl, anything will do) because you will want to generate the benchmark data automatically. I picked up Ruby as it's nowadays the language of my choice when it comes to any kind of Shell-like scripts. By choosing Ruby I have a large panel of classes to process benchmarking data, for instance I have a Benchmark class (wonderful), I have a CSV class (awfully documented, redundant), and I have a zillion of Gems for any kind of tasks I would need to do (although I always avoid them).

I first focused on retrieving the data I was interested into (memory, cpu time and file size) and saving it in the CSV format. By doing so I am able to produce charts easily with existing applications, and I was thinking maybe it was possible to use GoogleCL to generate charts from the command line with Google Docs but it isn't supported (maybe it will maybe it won't, it's up to gdata-python-client). However there is an actual Google tool to generate charts, it is the Google Chart API that works by providing a URI to get an image. The Google Image Chart Editor website helps you to generate the chart you want in a friendly WYSIWYG mode, after that it is just a matter of computing the data into shape for the URI. But well while focusing on the charts I found the Ruby Gem googlecharts that makes it friendly to pass the data and save the image.

Ruby Script

The Ruby script needs the following:
  • It was written with Ruby 1.9
  • Linux/Procfs for reading the status of processes
  • Googlecharts: gem install googlecharts
  • ImageMagick for the command line tool convert (optional)
The Ruby script takes a path as argument, with which it creates a tarball inside a tmpfs directory in order to avoid I/O latencies from a hard-drive. Next it runs a number of commands over the tarball from which it collects benchmark data. The benchmark data is then saved inside CSV files that are reusable within spreadsheet applications. The data is also reused to retrieve charts from the Google Chart API and finally it calls the ImageMagick tool ''convert'' to collect the charts inside a single image. The summary displayed on the standard output is also saved inside a text file.

The script is a bit long for being pasted here (more or less 300 lines) so you can download it from my workstation. If the link doesn't work make sure the web browser doesn't encode ~ (f.e. to "%257E"), I've seen this happening with Safari (inside my logs)! If really you are out of luck, it is available on Pastebin.

Benchmarks

The benchmarks are available for three kinds of data. Compressed media files, raw media files (image and sound, remember that the compression is lossless), and text files from an open source project.
Media Files
Does it make sense at all to compress already compressed data. Obviously not! Let's take a look at what happens anyway.


As you see, compression tools with focus on speed don't fail, they still do the job quick while gaining a few hundred kilo bytes. However the other tools simply waste a lot of time for no gain at all.

So always make sure to use a backup application without compression over media files or the CPU will be heating up for nothing.
Raw Media Files
Will it make sense to compress raw data? Not really. Here are the results:


There is some gain in the order of mega bytes now, but the process is still the same and for that reason it is simply unadapted. For media files there are existing formats that will compress the data lossless with a higher ratio and a lot faster.

Let's compare lossless compression of a sound file. The initial WAV source file has a size of 44MB and lasts 4m20s. Compressing this file with xz takes about 90s, this is very long while it reduced the size to 36MB. Now if you choose the format FLAC, which is doing lossless compression for audio, you will have a record. The file is compressed in about 5s to a size of 24MB! The good thing about FLAC is that media players will decode it without any CPU cost.

The same happens with images, but I lack knowledge about photo formats so your mileage may vary. Anyway, except the Windows bitmap format, I'm not able to say that you will find images uncompressed just like you won't find videos uncompressed... TIFF or RAW is the format provided by many reflex cameras, it has lossless compression capabilities and contains many information about image colors and so on, this makes it the perfect format for photographers as the photo itself doesn't contain any modifications. You can also choose the PNG format but only for simple images.
Text Files
We get to the point where we can compare interesting results. Here we are compressing data that is the most commonly distributed over the Internet.


Lzop and Gzip perform fast and have a good ratio. Bzip2 has a better ratio, and both LZMA and LZMA2 algorithms even better. We can use an initial archive of 10MB, 100MB, or 400MB, the charts will always look alike the one above. When choosing a compression format it will either be good compression or speed, but it will definitely never ever be both, you must choose between this two constraints.

Conclusion

I never heard about the LZO format until I wanted to write this blog post. It looks like a good choice for end-devices where CPU cost is crucial. The compression will always be extremely fast, even for giga bytes of data, with a fairly good ratio. While Gzip is the most distributed compression format, it works just like Lzop, by focusing by default on speed with good compression. But it can't beat Lzop in speed, even when compressing in level 1 it will be fairly slower in matter of seconds, although it still beats it in the final size. When compressing with Lzop in level 9, the speed is getting ridiculously slow and the final size doesn't beat Gzip with its default level where Gzip is doing the job faster anyway.

Bzip2 is noise between LZMA and Gzip. It is very distributed as default format nowadays because it beats Gzip in term of compression ratio. It is of course slower for compression, but easily spottable is the decompression time, it is the worst amongst all in all cases.

Both LZMA and LZMA2 perform almost with an identical behavior. They are using dynamic memory allocation, unlike the other formats, where the higher the input data the more the memory is allocated. We can see the evolution of LZMA is using less memory but has on the other hand a higher cost on CPU time. And we can see they have excellent decompression time, although Lzop and Gzip have the best scores but then again there can't be excellent compression ratio and compression time. The difference between the compression ratio of the two formats is in the order of hundred of kilo bytes, well after all it is an evolution and not a revolution.

On a last note, I ran the benchmarks on an Intel Atom N270 that has two cores at 1.6GHz but I made sure to run the compression tools with only one core.

A few interesting links:

Don’t produce Gzipped tarballs

A quick note so I can delete it from my desktop. In order to produce only a Bzip2 tarball with the Autotools, specially when running make distcheck, set the automake init call with these parameters:

AM_INIT_AUTOMAKE([no-dist-gzip dist-bzip2])

By the way I wonder if it's worth dumping bzip2 against xz.

Major changes in the Xfce Task Manager going 1.0

It's done. The task manager application available in Xfce for quite some years is now available with major changes. It has been rewritten from scratch, with GtkBuilder UI definitions and GObjects, everything is fresh and clean. The application has support for Linux, OpenBSD, FreeBSD and OpenSolaris.

Let's start by visual changes:
  • The buttons at the bottom are gone, the progress bars at the top are vanished, say hello to a toolbar with buttons and monitors.
  • You read well, monitors are in, they show a graph of the CPU and memory usage by time.
  • A status bar is visible at the bottom, it displays a general information about the system usage.
  • Icons are displayed beneath the task names.

Let's continue with less visual:
  • Tasks that start are displayed with a green background for a short delay and tasks that terminate with a red background.
  • Tasks which state is changing are temporarily displayed with a yellow background. This covers tasks changing their state from idle to running, vice versa and etc.
  • The tree views context menu contains the same actions as before, sending signals to the task and changing the priority. They have been polished however, for example the continue and stop signals aren't shown altogether anymore, and there are only five priorities to set ranging from Very low to Very high.
  • The tree view columns can be reordered as you wish.
  • An optional status icon can be activated allowing you to hide the application.
  • It is possible to display percentage values with more precision.
  • And finally, the default refresh rate is 750ms and it can be switched from 500ms up to 10s.

And the result is as follows:


The application is fully translated into fifteen languages!

Go to project webpage.

Meego installation on a USB stick

This post wouldn't be if the hard-drive from my Acer Aspire One didn't die. I have a fresh backup of the disk (a full 'dd' plus a separate one for just the home partition) so if I need something back, and I know I don't I care but backups are important, I can always mount it in a loopback and copy files.

The hard-drive is actually, what I want to call it, a cheap and fake SSD. It's a PATA SSD that I'm sure I will never find a replacement for. Look:


Now the dilemma was easy, or I threw the netbook to trash, or I found something to boot on. I started to look for a solution and Meego just got released, this is crazy timing. I downloaded the boot image and tried it out, and guess what, it is getting better and better. It's definitely more beautiful, it is getting faster, it has better dialogs for customization, well just try it out if you didn't yet, you wont be disappointed but surprised.

So in the end, installing a system on a USB stick is the only solution I can come up with. I ordered an extra USB stick, but mini please, a Kingston DTmini10! Now when I tell people this is my actual hard-drive, they are like “say-whaaat.”

The installation of Meego didn't went that fluently. I have two USB sticks, one with the boot image, another serving as target device for installation. The installation worked fine without any modification, it boots but ends on a black screen with the CAPS del blinking. Boo, kernel panic, or something else ungroovy. I also tried an installation with the file-system ext3, the default is btrfs, but then the grub installer fails and the Meego installer is knocked out in a waiting sequence. So I did a default installation again, sigh. After a search I tried out some parameters for the kernel command line and adding “rootdelay=8” did the trick. In fact, the USB stick boots without problem, but past that there is some delay for the kernel to discover the USB device, you can then see the following message:

   sd 11:0:0:0: [sdx] Assuming drive cache: write through

If there is no rootdelay parameter there is no root device found, and booting just ain't gonna work out. End of story. There are some tiny tweaks to be done afterwards. The kernel command line must point to the right root device, just like for the fstab file. The kernel command line can be edited in the file /boot/extlinux/extlinux.conf. Everything else works out just fine. Booting time, except the rootdelay, is acceptable, but shutting down seems to be endless, and precisely when I want the netbook to turn off I want it to be really fast. I'm going to send it to sleep more often than usual, by closing and opening the lid, which is the fastest “boot” sequence one can get ;-)


Update: I've been wrong stating the shutdown process is taking ages, I just did a shutdown and this time it went quick, so something must have been be unlucky and the disk synced something around and around.

Meego installation on a USB stick

This post wouldn't be if the hard-drive from my Acer Aspire One didn't die. I have a fresh backup of the disk (a full 'dd' plus a separate one for just the home partition) so if I need something back, and I know I don't I care but backups are important, I can always mount it in a loopback and copy files.

The hard-drive is actually, what I want to call it, a cheap and fake SSD. It's a PATA SSD that I'm sure I will never find a replacement for. Look:


Now the dilemma was easy, or I threw the netbook to trash, or I found something to boot on. I started to look for a solution and Meego just got released, this is crazy timing. I downloaded the boot image and tried it out, and guess what, it is getting better and better. It's definitely more beautiful, it is getting faster, it has better dialogs for customization, well just try it out if you didn't yet, you wont be disappointed but surprised.

So in the end, installing a system on a USB stick is the only solution I can come up with. I ordered an extra USB stick, but mini please, a Kingston DTmini10! Now when I tell people this is my actual hard-drive, they are like “say-whaaat.”

The installation of Meego didn't went that fluently. I have two USB sticks, one with the boot image, another serving as target device for installation. The installation worked fine without any modification, it boots but ends on a black screen with the CAPS del blinking. Boo, kernel panic, or something else ungroovy. I also tried an installation with the file-system ext3, the default is btrfs, but then the grub installer fails and the Meego installer is knocked out in a waiting sequence. So I did a default installation again, sigh. After a search I tried out some parameters for the kernel command line and adding “rootdelay=8” did the trick. In fact, the USB stick boots without problem, but past that there is some delay for the kernel to discover the USB device, you can then see the following message:

   sd 11:0:0:0: [sdx] Assuming drive cache: write through

If there is no rootdelay parameter there is no root device found, and booting just ain't gonna work out. End of story. There are some tiny tweaks to be done afterwards. The kernel command line must point to the right root device, just like for the fstab file. The kernel command line can be edited in the file /boot/extlinux/extlinux.conf. Everything else works out just fine. Booting time, except the rootdelay, is acceptable, but shutting down seems to be endless, and precisely when I want the netbook to turn off I want it to be really fast. I'm going to send it to sleep more often than usual, by closing and opening the lid, which is the fastest “boot” sequence one can get ;-)


Update: I've been wrong stating the shutdown process is taking ages, I just did a shutdown and this time it went quick, so something must have been be unlucky and the disk synced something around and around.

Moblin blazing fast

I updated my netbook to give it a new look. I switched the Xfce Panel against bmpanel2 and changed the background (the previous definitelly lasted very long.) Not much changes, but I topped a cold boot of about six seconds, always faster baby :-P And the window manager is OpenBox by the way.


The only real useful entry missing in this panel is a battery monitor. At least I have an indicator over the keyboard that starts blinking when there is about three percents left. What I like about this panel is the cool themes that it is provided with, however the configuration is set through a hand-written configuration file which sucks but what do you want, it is extremely lightweight on the other hand.

Update: Should I mention I totally forgot about the Xfce power manager? Well I did, and it is provided with a notification icon displaying the battery status :-) However I had to fix the default ACPI script related to the lid, since HAL doesn't list it, in order to get the netbook to go into sleep.

Moblin blazing fast

I updated my netbook to give it a new look. I switched the Xfce Panel against bmpanel2 and changed the background (the previous definitelly lasted very long.) Not much changes, but I topped a cold boot of about six seconds, always faster baby :-P And the window manager is OpenBox by the way.


The only real useful entry missing in this panel is a battery monitor. At least I have an indicator over the keyboard that starts blinking when there is about three percents left. What I like about this panel is the cool themes that it is provided with, however the configuration is set through a hand-written configuration file which sucks but what do you want, it is extremely lightweight on the other hand.

Update: Should I mention I totally forgot about the Xfce power manager? Well I did, and it is provided with a notification icon displaying the battery status :-) However I had to fix the default ACPI script related to the lid, since HAL doesn't list it, in order to get the netbook to go into sleep.

VLC with GTK+ look-n-feel

To get Qt applications to look like GTK+ applications run qtconfig and in Select GUI Style choose GTK+. Next click in the menu bar File > Save.

Something is still puzzling me, why does GNOME run VLC automatically with native GTK+ look-n-feel and not Xfce?

Update: Thanks to the power of tig and grep, I figured the Qt library (qt_init function) defines the desktop environment as GNOME for Xfce (this results in GTK+ theming, GNOME like Open dialogues, etc) by retrieving an X11 atom on the root window and compares it to “xfce4” but it seems that this doesn't work nowadays (at least it didn't work within an Xfce 4.7 desktop session). I'm looking forward for sending patches.

Update2: The latest version Qt 4.6.2 doesn't include the code for checking the X11 atom (it's in git), which explains why it doesn't work.

VLC with GTK+ look-n-feel

To get Qt applications to look like GTK+ applications run qtconfig and in Select GUI Style choose GTK+. Next click in the menu bar File > Save.

Something is still puzzling me, why does GNOME run VLC automatically with native GTK+ look-n-feel and not Xfce?

Update: Thanks to the power of tig and grep, I figured the Qt library (qt_init function) defines the desktop environment as GNOME for Xfce (this results in GTK+ theming, GNOME like Open dialogues, etc) by retrieving an X11 atom on the root window and compares it to “xfce4” but it seems that this doesn't work nowadays (at least it didn't work within an Xfce 4.7 desktop session). I'm looking forward for sending patches.

Update2: The latest version Qt 4.6.2 doesn't include the code for checking the X11 atom (it's in git), which explains why it doesn't work.

New notes plugin release 1.7.3

Three months since the last release, and three months since it is available as a separate standalone application running in the notification area. This has made it a lot easier to test and debug, as before I had to build the plugin, install the plugin, restart the panel or remove/readd the plugin in the panel, now I just have to run ./xfce4-notes from the source directory.

This new release has seen some structural tree changes to save time during compilation. Now everything is in src/ and lib/, where lib/ contains code to build an XnpHypertextView, an XnpNote (a composite-widget that embeds a GtkScrolledWindow with an XnpHypertextView and sends “save” signals on changes), an XnpWindow with the custom made navigation and title bars and the right click menu on the title bar, and finally an XnpApplication class that is the heart of everything, it handles creations/deletions of notes, loads/saves the data, etc. The src/ directory contains the main files for the panel plugin, the status icon, the popup command and the settings dialogue.

The new stuff is mostly eye-candy as stated in the previous blog entry. The GTK+ RC style has been pimped up with custom made scrollbars and the source code contains a self-drawn close button. The stuff about GTK+ scrollbars theming is grossly explained on live.gnome.org but I opened the GTK+ Dust theme files which was, to me, more understandable :-) Also it was because of this particular theme I took a look at customizing the scrollbars, see below the before/after screenshots. The older article about writing a Widget with Cairo helped me getting started from scratch with an empty “close button” widget to replace the simple GtkButton with label. As I liked very much the time passed on these changes I contributed a tutorial “Monochrome icon” available only in PDF as of today which I hope to be useful for Vala beginners but also a nice update of the article about Cairo but with Vala language.


The fixes included in this release are the following: correctly restore sticky-window and keep-above states after some race conditions, and restore tab label orientation after renaming a note. And last but definitely not least the undo feature was not working because an internal timeout wasn't reset to zero which made the code think a snapshot was needed and thus the undo/redo buffers ended with the same content after the timeout elapsed. Thanks to Christian (the developer behind Midori) otherwise I would still not have taken a look around this!

The forthcoming features I have in mind would be a search dialogue and per-note options for activating a stripped down “markdown” syntax, an orthographic corrector and wrapping words which is the default for the moment.

The release is available at archive.xfce.org.

Thanks for the feedbacks and reports you sent and will send back.

Update: The tutorial is now also available on the Xfce wiki.