Xfce

Subdomains
 

Eatmonkey 0.1.3 benchmarking

  • February 13, 2010
  • Mike Massonnet
Eatmonkey has now been released for the 4th time and I started to use it to download videos from FOSDEM2010 by drag-n-dropping the links from the web page to the manager :-)

I downloaded four files and while they were running I had a close look at top and iftop to monitor the CPU usage and the bandwidth usage between the client/server (the connection between eatmonkey and the aria2 XML-RPC server running on the localhost interface).

I had unexpected results and was surprised by the CPU usage. It is very high currently which means I have a new task for the next milestone, getting the CPU footprint low. The bandwidth comes without surprises, but since the milestone will target performance where possible I will fine down the number of requests made to the server. This problem is also noticeable in the GUI in that it tends to micro-freeze during the updates of each download. So the more active downloads will be running the more the client will be freezing.

Some results as it will speak more than words:
Number of active downloadsReceptionEmissionCPU%
4 downloads144Kbps18Kbps30%
3 downloads108Kbps14Kbps26%
2 downloads73Kbps11Kbps18%

I will start by running benchmarks on the code itself, and thanks to Ruby there is built-in support for Benchmarking and Profiling. It comes with at least three different useful modules: benchmark, profile and profiler. The first measures the time that the code necessitated to be executed on the system. It is useful to measure different kind of loops like for, while or do...while, or for example to see if a string is best to be compared through a dummy compare function or via a compiled regular expression. The second simply needs to be included at the top of a Ruby script and it will print a summary of the time passed within each method/function call. The third does the same except it is possible to run the Profiler around distinctive blocks of code. So much for the presentation, below are some samples.

File benchmark.rb:
#!/usr/bin/ruby -w

require "benchmark"
require "pp"

integers = (1..10000).to_a
pp Benchmark.measure { integers.map { |i| i * i } }

Benchmark.bm(10) do |b|
b.report("simple") { 50000.times { 1 + 2 } }
b.report("complex") { 50000.times { 1 + 2 - 6 + 5 * 4 / 2 + 4 } }
b.report("stupid") { 50000.times { "1".to_i + "3".to_i * "4".to_i - "2".to_i } }
end

words = IO.readlines("/usr/share/dict/words")
Benchmark.bm(10) do |b|
b.report("include") { words.each { |w| next if w.include?("abe") } }
b.report("regexp") { words.each { |w| next if w =~ /abe/ } }
end

File profile.rb:
#!/usr/bin/ruby -w

require "profile"

def factorial(n)
n > 1 ? n * factorial(n - 1) : 1;
end

factorial(627)

File profiler.rb:
#!/usr/bin/ruby -w

require "profiler"

def factorial(n)
(2..n).to_a.inject(1) { |product, i| product * i }
end

Profiler__.start_profile
factorial(627)
Profiler__.stop_profile
Profiler__.print_profile($stdout)
Update: The profiling showed that during a status request 65% of the time is consumed by the XML parser. The REXML class is written 100% in Ruby, and that gives a good hint that the same request done with a parser written in C may present a real boost. On another hand, the requests are now only run once periodically and cached inside the pooler. This means that the emission bitrate is always the same and that the reception bitrate grows as there are more downloads running. And as a side-effect there is less XML parsing done thus less CPU time used.

Eatmonkey 0.1.3 benchmarking

  • February 13, 2010
  • Mike Massonnet
Eatmonkey has now been released for the 4th time and I started to use it to download videos from FOSDEM2010 by drag-n-dropping the links from the web page to the manager :-)

I downloaded four files and while they were running I had a close look at top and iftop to monitor the CPU usage and the bandwidth usage between the client/server (the connection between eatmonkey and the aria2 XML-RPC server running on the localhost interface).

I had unexpected results and was surprised by the CPU usage. It is very high currently which means I have a new task for the next milestone, getting the CPU footprint low. The bandwidth comes without surprises, but since the milestone will target performance where possible I will fine down the number of requests made to the server. This problem is also noticeable in the GUI in that it tends to micro-freeze during the updates of each download. So the more active downloads will be running the more the client will be freezing.

Some results as it will speak more than words:
Number of active downloadsReceptionEmissionCPU%
4 downloads144Kbps18Kbps30%
3 downloads108Kbps14Kbps26%
2 downloads73Kbps11Kbps18%

I will start by running benchmarks on the code itself, and thanks to Ruby there is built-in support for Benchmarking and Profiling. It comes with at least three different useful modules: benchmark, profile and profiler. The first measures the time that the code necessitated to be executed on the system. It is useful to measure different kind of loops like for, while or do...while, or for example to see if a string is best to be compared through a dummy compare function or via a compiled regular expression. The second simply needs to be included at the top of a Ruby script and it will print a summary of the time passed within each method/function call. The third does the same except it is possible to run the Profiler around distinctive blocks of code. So much for the presentation, below are some samples.

File benchmark.rb:
#!/usr/bin/ruby -w

require "benchmark"
require "pp"

integers = (1..10000).to_a
pp Benchmark.measure { integers.map { |i| i * i } }

Benchmark.bm(10) do |b|
b.report("simple") { 50000.times { 1 + 2 } }
b.report("complex") { 50000.times { 1 + 2 - 6 + 5 * 4 / 2 + 4 } }
b.report("stupid") { 50000.times { "1".to_i + "3".to_i * "4".to_i - "2".to_i } }
end

words = IO.readlines("/usr/share/dict/words")
Benchmark.bm(10) do |b|
b.report("include") { words.each { |w| next if w.include?("abe") } }
b.report("regexp") { words.each { |w| next if w =~ /abe/ } }
end

File profile.rb:
#!/usr/bin/ruby -w

require "profile"

def factorial(n)
n > 1 ? n * factorial(n - 1) : 1;
end

factorial(627)

File profiler.rb:
#!/usr/bin/ruby -w

require "profiler"

def factorial(n)
(2..n).to_a.inject(1) { |product, i| product * i }
end

Profiler__.start_profile
factorial(627)
Profiler__.stop_profile
Profiler__.print_profile($stdout)
Update: The profiling showed that during a status request 65% of the time is consumed by the XML parser. The REXML class is written 100% in Ruby, and that gives a good hint that the same request done with a parser written in C may present a real boost. On another hand, the requests are now only run once periodically and cached inside the pooler. This means that the emission bitrate is always the same and that the reception bitrate grows as there are more downloads running. And as a side-effect there is less XML parsing done thus less CPU time used.

Backward compatibility for Ruby 1.8

  • February 6, 2010
  • Mike Massonnet
As I'm currently writing some Ruby code and that I started with version 1.9 I felt onto cases where some methods don't exist for Ruby 1.8. This is very annoying and I started by switching the code to 1.8 method calls. I disliked this when it came to Process.spawn which is a one line call to execute a separate process. Rewriting it takes around 5 lines instead.

So I had the idea to reuse something I already saw once. I write a new file named compat18.rb and include it within the sources that need it. Ruby makes it very easy to add new methods to existing classes/modules anyway, even if they exist already, so I just did it and it works like a charm.

Here is a small snippet:
class Array
        def find_index(idx)
                index(idx)
        end
end

class Dir
        def exists?(path)
                File.directory?(path)
        end
end

Update: It can happen that a fallback method from Ruby 1.8 has been totally dropped and replaced against a new method in 1.9, and in this case the older method has to be checked if it exists, and otherwise make a call to the parent.
class Array
        def count
                if defined? nitems
                        return nitems
                else
                        return super
                end
        end
end

The download manager is in the wild

  • January 24, 2010
  • Mike Massonnet
So it's finally done, it took very long, but it's done. The download manager I once had in mind is taking off into the wildness :-) Of course it took long because I never did something with it, writing a front-end to wget/curl isn't interesting -- who cares about downloading HTTP/FTP files when the web browser handles it for you anyway -- and reusing GVFS doesn't make sense cause really you don't want to download from your trash:// or whatever proto:// and again only HTTP/FTP is not interesting. Not at all. I have come across Uget and other very good projects but most of them are either writing the code to handle the protocol like HTTP and/or are looking forward to handle more interesting protocols like BitTorrent. I think it's a very tough job that demands too much for a one-maintainer project. Recently I saw the new release of aria2 that comes with an XML-RPC interface and this took all my interest during 4 days. I believe this utility is very promising and I had really like to write the good and user-friendly XML-RPC GUI client that it seems to be missing!

What is so exciting about aria2? In case you know the project you don't have to read, but it is worth mentionning the features of this small utility. It supports HTTP(s)/FTP but also BitTorrent and Metalinks. It is widely customizable for each specific protocol. It can download one file by splitting it into several pieces and using multiple connections and even mix HTTP URIs with BitTorrent and by the same time upload to BitTorrent peers what has been downloaded through HTTP. So this has to be the perfect candidate to write a nice download manager, hasn't it?

The client is a very first version that I intended to code name draft although the release assistant on xfce.org doesn't allow this. Instead it will take the more neutral road of 0.1.0 to 0.1.1 etc until 0.2.0 followed by stable fix releases.

Why draft? Simple. It's being written with a higher level language than C but not even Vala :-) High-level languages are a great deal when starting a new application, as you can type more and get more, instead of typing like a dog for a rocking hot, well lousy, window. Since I do like Ruby, it's being written in Ruby currently, and it depends on the ruby-gnome2 project for the bindings. To get a picture, a main file to open a window takes 3 lines. Of course the final version is meant to be written in Vala/C, but I still need to convince myself that Vala+libsoup isn't an option that is going to waste too much time. Also at first glance libsoup looks easy to use, it allows to build XML-RPC requests, to request the HTTP bodies and to send messages, but it is not an XML-RPC client and you never know how well the Vala bindings will play. This means extra attention for small things. Starting an application from scratch with such constraints are usually a big time-killer therefore using like in this case an existing XML-RPC client is very important. The GUI is done with Glade in GtkBuilder format and reusing it into a new language will be pretty easy.


So what's next? I'll just wait for some feedback see what the audience thinks about it, if at all, and polish here and there. Keep tuned for the next update.