This is helloworld spped test... so it shows nothing about speed other than mentioned in comments.
For all that doesn't like my test: For me it shows how small applications can work and if they are smart enough to ignore dummy code :) I've figured it after reading comments.
Summary:
Java is slow,
Scala works probably the same but starts longer.
Groovy is slower.
Groovy++ - maybe some day I will have time to test it.
I read about Groovy and Scala, both languages are interesting. Groovy is easier and have much better support, it is like heaven and hell when you compare Scala and Groovy support (Intelli IDEA is only one that works for me... but I think that when I install in Eclipse Groovy plugin it breakes Scala support, Groovy and Scala are fighting in my computer and when one works second breaks, but it is only for me).
I like Groovy but i read about how slow it is, lets write veeery simple test and check it:
Scala:
object Sample{
def main(args: Array[String]) = {
for( g <- 1 to 10) { test()}
}
def test() = {
val startTime = System.nanoTime();
for(i <- 1 to 100){
for(j <- 1 to 100){
for(k <- 1 to 100){
for(l <- 1 to 100){
i+j+k+l
}
}
}
}
val time = System.nanoTime() - startTime
System.out.println(time/1000000);
}
}
Response:
4731
4729
4661
4871
4765
4618
4722
4805
5004
4768
Groovy
public class Sample{
static main(args){
for(i in 1..10){ test()}
}
static test(){
long startTime = System.nanoTime();
for(i in 1..100){
for(j in 1..100){
for(k in 1..100){
for(l in 1..100){
i+j+k+l
}
}
}
}
long time = System.nanoTime() - startTime
System.out.println((int)time/1000000);
}
}
Response:
16077
15462
16454
17798
17725
17814
16148
16081
16164
22489
Java:
public class Main {
public static void main(String[] args){
for(int g = 0; g<10; g++){ test(); }
}
static void test(){
long startTime = System.nanoTime();
for(int i = 1; i< 100; i++){
for(int j = 1; j< 100; j++){
for(int k = 1; k< 100; k++){
for(int l = 1; l< 100; l++){
int m = i+j+k+l;
}
}
}
}
long time = System.nanoTime() - startTime ;
System.out.println(time/1000000);
}
}
Response:
191
175
170
184
182
173
176
168
168
169
-
-
All are from Intelli IDEA, as "Run Application".
-
Bo be honest with Scala and Groovy I compiled them from command line and run compiled versions:
Scala:
4372
4264
4595
4571
4313
4378
4369
4335
4458
4588
Groovy:
15854
15728
16917
17265
16141
16504
16155
16849
16007
16496
-
Faster, but still so slow...
Conclusion:
-
4731 - Scala
16454 - Groovy
176 - Java
From compiled versions:
-
15854 is 15,9 seconds...
-
4588 is 4,6 seconds
-
176 is so fast! ;) 0,176 s.
With some changes in Java code I get about 180-200.
If I have errors in code please point them in comments.
I must think about that numbers, maybe I have errors in code?
I am waiting to responses with yours tests!
Regards,
Krzysztof Kula
Language comparison tests are one of the hardest problems and writing a good test is not that simple. And there is no one ultimate, good tests to compare two languages.
ReplyDeleteYou always have to take into account what you actually want to test, operations and application type.
In my opinion your test is not very accurate. The variable inside the internal loop is not used anywhere hence a good compiler may just remove the internal expression significantly increasing the loop speed.
I don't know if any compilers for the above languages actually do that but if at least one do it (Java for example) then the above test compares compilers instead of the language.
And this is not all. Please note, I do not know much about Scala but I know that Groovy doesn't use primitive types (long, int, boolean, arrays, etc...) instead it performs all operations on Objects (Long, Integer, Boolean, ArrayList, etc...). This causes the obvious slow down related to object allocation and obvious slower operations on objects than primitives.
It also causes non-obvious slowdown related to much higher GC activity. The thing is that this GC activity can kick out much later, depending on how much memory you assigned to JVM it may even become apparent after a few minutes when the test is long over.
Therefore the same test run over extended period of time may show different results from the test run over a few seconds only.
Again, this is just a matter what really you are interested in. The client applications usually don't run for a very long time and GC activity normally doesn't matter. For server applications this is a different story.
I think an interested modification to the test would be saving calculation results from the internal loop in some cyclic array of 1000 elements size. This way you could make sure the expression is actually executed and also you could add some array operations to the test. It seems to me that running each test for a couple of minutes could be interesting as at some point GC activity should start affecting the test and may slow down some runs.
Interesting what You write! Yes variable can be replaced by compilator, by in real time in code before/afrer many changes and bugfixes in a long time that unused code can live and slow down application. It is pleased to know that my compiler is smart enougth to remove it ;)
ReplyDeleteAbout GC, yesterday I made myself a 'free day', and I don't know when I done another one. So If you done some comparision with Memory I will be interested to read it :)
In title of my post is "Speed Test" and even with GC all values of code execution always must be averaged so in the end I think that results must me al least similar.
About int and Integer in Groovy, as I know there is no option in Groovy to use int type. In this test I was trying to test speed with code similar to that I will write in real life, so I don't bother in optimising.
If You do some test please get me know :)
Comparing Scala'a powerfull for comprehension to simple loop is just stupid. While loop will be mus faster.
ReplyDeleteBut, you have no fracking idea how to do benchmarks. HotSpot will remove all those stupid loops. So Java time will be just 0 (zero). LOL
http://www-128.ibm.com/developerworks/java/library/j-jtp02225.html?ca=drs-j0805
ReplyDeleteSo don't spread misinformation please...
Where are about run params? Ok. Truth is that You know nothing about how JVM works. First of all, when Running Your test with -server JVM option the results for Scala are:
ReplyDelete969
745
747
736
745
745
744
725
748
743
But for in Scala is something completely different than in Java.In Scala for is a Range that's why it's slower.
here's upgraded version of Your "test" :
def test() = {
val startTime = System.nanoTime();
var i, j, k, l = 0
while(i < 10000) {
i+=1
while(j < 10000){
j+=1
while(k < 10000){
k+=1
while(l < 10000){
l+=1
i+j+k+l
}
}
}
}
Now note that I multiply Your funny 100 by 100, and here're results:
1
1
1
1
1
1
1
1
8
0
So please. Don't compare such garbage language like Groovy with Scala, and don't spread disinformation with such sloppy "benchmarks".
Btw. I'm running MacBook pro with c2d 2.4Ghz, and I'm using Idea too.
Wow, we're just setting up a site ihavenoideahowtobenchmark.org, be sure you're gonna be there.
ReplyDelete@dmilith About Scala results: Are You fanboy? Don;t get me wrong but that You proove only that 'while' is faster than 'for'.
ReplyDeleteI never think of myself as an expert and that blog is not a place to share with the world with my thougths, and to learn from comments ;)
@dmilith that garbage Groovy language with code like that yours:
public class Sample{
static main(args){
for(i in 1..10){ test()}
}
static test(){
long startTime = System.nanoTime();
int i = 0, j=0, k=0, l=0, sum = 0;
while(i < 100){
i+=1;
while(j < 100){
j+=1;
while(k < 100){
k+=1;
while(l < 100){
l+=1;
i+j+k+l;
}
}
}
}
long time = System.nanoTime() - startTime
System.out.println((int)time/1000000);
}
}
Result:
226
3
9
8
0
0
0
0
0
0
So only first loop in Groovy is slower. And nest results are successively faster.
The reason of doing that simple test was to choose a language to learn, so I can tell that I have only basic knowledge about that languages! Language speed is important to me so I write those very simple tests.
Java version of your code has:
0
0
0
0
0
0
0
0
0
0
Can You tell me why in Scala for loop is a Range? I often use for loops and want them to be fast and efficient.
But ofter these test I probably start using while loops ;)
While I said "garbage language" I did want to say that Groovy has mess in code and ideology of whole language.
ReplyDeleteOfcourse Groovy will have same result.. Like Anonimowy said before, hotspot will remove all those loops.. No matter which JVM based language You'll use.. It will return similar results every time.
Note that for loops has just different purpose in Scala. They're more powerful than in Java for example.
And no I'm not fanboy. I know some weak sides of Scala (f.e. boxing of primitives, incompatibility with List and Iterators from Java which is caused different purpose of Lists in Scala, etc.) but in comparison with rest of JVM languages it's the best choice for productive and effective programming
(It's my opinion ofcourse)
Your code is not counting like mine, try to fix it and then post results :)
ReplyDeleteHint: what about values of j,k,l after first loop? ;)
I still cant decide... Maybe I will learn only additional Java library instead of new Language. Even so I think about : Groovy, Scala, Python, Ruby, C#(I know that it is gaining popularity and friend told me that ti is used in AGH for students projects).
Na AGH zwykle nie ma przymusu użycia konkretnego języka, więc nie wiem co to za argument. To samo mogę powiedzieć o Scali, Pythonie czy Ruby.
ReplyDeleteŁatwiej rzucić kostką niż wybierać język na podstawie zagnieżdżonych pętli (zwłaszcza że tylko z pozoru porównujesz to samo), rezultat będzie podobny.
Chodziło mi raczej o przeprowadzenie testu na własną rękę którym by mi dawał jakieś orientacyjne pojęcie jak z tą szybkością jest. Czytałem ostatnio o tym jaki Groovy jest wolny postanowiłem to sprawdzić na własną rękę, nie nadaję temu nie-wiadomo-jakiej wagi i ostateczności :)
ReplyDeleteCo do C# to muszę dopytać dla pewności ale kolega jest na uzupełniających magisterskich i ma projekty np. w Javie, ASP(C#) jako że on nie zna żadnego z tych języków na 95% mogę powiedzieć że takiej dowolności nie miał, no może mógł pisać w Javie albo w C# ;)
Scala ma wydajność prawie taką samą jak Java (czasem trochę wolniej, czasem trochę szybciej), a ty się naucz wpierw pisać mikrobenchmarki bo porównywanie pustych pętli jest bez sensu (HotSpot je wytnie i ten kod się po prostu nie wykona). Także porównywanie form comprehension z prymitywną pętlą Javy jest kompletnie bez sensu. Scalowe "for" to dużo więcej niż pętla.
ReplyDeleteŻaden tam Groovy, Python, Ruby czy PHP nie podkoczy wydajnościowo do Scali, to rzecz oczywista. Co do Grooviego to jego twórca sam powiedział, że gdyby wcześniej poznał Scalę to Groovy by nigdy nie powstał. :-D Zresztą po co pisać w Groovy? Dla Grails? Toż to pokraczna kopia Rails. Co innego Lift (napisany w Scali), ale to wyższa poprzeczka.
W tym wszystkim nie wspomniano o bardzo potężnym i ciekawym języku jakim jest Clojure. Jego potęga to potęga Lispa gdzie dane są kodem a kod danymi (Clojure jest homoiconic). Poza tym Clojure jest też najwydajniejszym, dynamicznie typowanym językiem dla JVM. Ogólnie polecam doskonała prezentację http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey oraz 2 czesc: http://www.infoq.com/presentations/Value-Identity-State-Rich-Hickey. Takie prezentacje trochę otwierają oczy. Paul Graham miał rację pisząć, że tylko ci co poznali najpotężniejszy z języków programowania są w stanie porównywać inne języki. Reszta nie wie po prostu o czym mówi. Obowiązkowa lektura: "Beating the Averages" http://www.paulgraham.com/avg.html
'while' is faster than 'for' - We need to reinvent the whole of computer science again.
ReplyDeleteNiedawno kończyłem AGH i projekty pisałem praktycznie w dowolnym języku, takim który był wg mnie najodpowiedniejszy (czyt. najszybciej było skończyć projekt) do mojego zadania. Problemy były jedynie wtedy kiedy prowadzący nie chciał zestawić środowiska dla mojego projektu, więc albo musiałem o to zadbać sam (głównie wykorzystując laptop) albo zrezygnować i pisać w języku pod który infrastruktura była już przygotowana. Tak naprawde mało kogo obchodził kod, dlatego ta dowolność technologii/języka.
ReplyDeleteCo do benczmarku to wszystko byłoby OK gdybyś testował te same rzeczy, bo nawet microbenchmark daje jakieś wyniki do interpretacji. Ale w obecnej sytuacji wprowadzasz w błąd siebie i innych.
Widzę, że kolega Krzychu nie tylko nie ma pojęcia o Scali ani o benchmarkowaniu Javy ale także nie za bardzo umie pisać w Javie. HotSpot dla Javy, podobnie jak dla Scali, musi tu zwrócić zera, bo te pętle zostaną po prostu wyrzucone z bytecodu.
ReplyDeletehttp://gist.github.com/282504#file_dupa_nie_test.java
I have a nasty feeling that you have no idea how to write a benchmark test and that your knowledge of Scala is really poor. This post ist just a rubbish.
ReplyDelete