Ruby vs JRuby Fractal Benchmark
After reading about benchmarking various languages generating fractal patterns, I tried throwing the Ruby code at both the standard MRI Ruby and JRuby interpreters to compare relative performance. This is all totally unscientific, but is interesting all the same.
MRI Ruby (Matz Ruby Interpreter)
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Rendering
Ruby Elapsed 6.732136
JRuby
ruby 1.8.5 (2007-06-04 rev 3812) [i386-jruby1.0.0RC3]
Rendering
Ruby Elapsed 68.757000
C Ruby is the clear winner, but that’s not so surprising. What is surprising is how far JRuby was behind - by at least 10 times. I’ve heard good things about JRuby’s ability to compile Ruby down to bytecode, but I didn’t have too much luck with that. Admittedly I didn’t play around with it for long.
I was curious where the bottlenecks are in both versions. After running the code through both VMs with the -r profile flag, really slow floating point operations turned out to be the culprit. The profile for each was pretty much identical. Lots of float operations (+ / < * etc), with everything multiplied by 10 for the JRuby VM.
So what does this tell us? Pretty much nothing. Most apps won’t be doing this kind of CPU heavy crunching (and if you’re using an interpreted language you’re asking for trouble). Especially the kind of apps you’d use either MRI Ruby or JRuby for, which spend most of their time waiting for the database. I might have another play and see if I can get JRuby to run using bytecode - the hype is that it will be as fast if not faster than MRI.
Disclaimer - I’m actually a huge fan of both MRI Ruby and JRuby and just did this out of curiosity. Put AWAY the flame torches now…
Update - JRuby can actually be as fast or faster than MRI
Charles Oliver Nutter (of the JRuby team), has added some suggestions for tuning the JRuby VM. The following is also in the comments, but I’ve included it in the main post as it is useful information that might get missed. What is really interesting is that JRuby is faster in this case when the VM is tuned and the code gets a chance to run enough for the JIT compiler to kick in.
FYI, I grabbed the code in question to try it out myself. These are my numbers using JRuby trunk (not really different from RC3), Java 6 server VM on OS X, and the following command line:
jruby -J-server -J-Djruby.jit.threshold=0 -O fractal.rb
JRuby:
Ruby Elapsed 6.454000Ruby 1.8.6:
Ruby Elapsed 6.971203The command line above adds three options:
-J-server turns on the server VM
-J-Djruby.jit.threshold=0 sets the JIT threshold to zero, so it will try to compile methods immediately (critically important for this benchmark since the methods are only called once), and -O turns off ObjectSpace, which is pure overhead for us.
Additionally, if I let the benchmark run ten times, allowing the JVM to optimize it further, I get the following result for JRuby on the tenth iteration:
Ruby Elapsed 5.629000
There’s a good page in the JRuby Wiki for performance tuning JRuby. The reason this wasn’t compiling is twofold: the methods weren’t called enough times to trigger the JIT, and the script itself had a “class” definition in it, which the compiler doesn’t yet handle.
Cheers Charles, extremely useful.
June 7th, 2007 at 2:05 am
I’d guess the code in question simply isn’t compiling in JRuby. JRuby only compiles about 50% of methods out there, and a substantially smaller percentage of actual code (since small methods usually compile and large ones usually don’t). We haven’t focused on completing the compiler for 1.0, but after the release we’ll hit it a lot harder. We’ve got a lot of planned optimizations coming.
You might get a boost if you make sure you’re running under Java 6 with the server VM…but if it’s this slow for you right now there’s probably some bottleneck in JRuby artificially slowing it down.
June 7th, 2007 at 2:24 am
FYI, I grabbed the code in question to try it out myself. These are my numbers using JRuby trunk (not really different from RC3), Java 6 server VM on OS X, and the following command line:
jruby -J-server -J-Djruby.jit.threshold=0 -O fractal.rb
JRuby:
Ruby Elapsed 6.454000
Ruby 1.8.6:
Ruby Elapsed 6.971203
The command line above adds three options:
-J-server turns on the server VM
-J-Djruby.jit.threshold=0 sets the JIT threshold to zero, so it will try to compile methods immediately (critically important for this benchmark since the methods are only called once), and -O turns off ObjectSpace, which is pure overhead for us.
Additionally, if I let the benchmark run ten times, allowing the JVM to optimize it further, I get the following result for JRuby on the tenth iteration:
Ruby Elapsed 5.629000
There’s a good page in the JRuby Wiki for performance tuning JRuby. The reason this wasn’t compiling is twofold: the methods weren’t called enough times to trigger the JIT, and the script itself had a “class” definition in it, which the compiler doesn’t yet handle.
June 7th, 2007 at 6:44 am
um, holy cow? JRuby has practically zero performance penalty?
June 7th, 2007 at 8:51 am
Excellent info Charles. I’ve included your 2nd comment in the main post.
June 8th, 2007 at 12:17 am
Just for reference, in Java (1.5 u9) it takes 0.07 seconds in my machine after the second iteration. which is around 100 times faster i think.. well, sure it is a micro benchmark .
June 8th, 2007 at 4:03 am
Java benefits from using actual primitive values, so that’s not quite a fair comparison to Ruby’s everything-is-an-object. But yes, we have a lot more work to do. There may even be a chance we can use primitives in certain situations.
June 11th, 2007 at 3:47 am
Julian Doherty wrote What is really interesting is that JRuby is faster in this case when…
and when we ignore jvm startup by measuring elapsed time within the program…
What is really interesting is that performance measurements come bundled with implicit usage scenarios which may or may not be valid.
(Those JRuby guys are doing great work!)
July 6th, 2007 at 8:34 am
Might it be a good idea to give a lot of credit for the performance to the Java virtual machine.
I’m seeing “Ruby Virtual Machine”, but I’d like to see the actual JVM be given some cudos. Nothing wrong with taking some pride in all this hard work…