Free/Open Cross-Platform SPEC Substitutes?

ErnstKompressor asks: "In light of the endless debate regarding top-dog performance between all flavors of OS/Hardware, the failings inherent in the various methods of testing such diverse systems, and the cost of existing software solutions, I was wondering what the Slashdot community recommends in the way of free, open-source, cross-platform, performance evaluation software. Additionally, how should one go about testing systems to obtain the fairest results? What compilers are evenly matched amongst different hardware? What balance should one strive for regarding optimizations and platform specific enhancements versus results that represent real-world performance? Finally, should such tests take into account the sub-systems available, such as 3D performance and the various Quake-FPS metrics?"

Already under way...

[Anonymous Coward]

Some of the programmer Arsians have already started work on the Ars TestBench 2.0 [sourceforge.net].

It's a Sourceforge project and slated to be GPL.

The goal is to produce a valid, relevant, cross-platform benchmark that doesn't cost thousands of dollars.

The kernel says...

[More Karma Than God]

Just compile the Linux kernel.

LMbench

[Shewmaker]

LMbench [bitmover.com] is great for testing the subsystems of different UNIX systems. It is probably one of the most useful benchmarks other than whatever applications you will actually be running.

Not a single number

[Carnildo]

The biggest problem with most benchmarks is that they try to reduce system performance to a single number. A good benchmark would produce a set of numbers, one for each subsystem such as integer performance, floating-point performance, logic performance, 2D graphics performance, and 3D graphics performance.

I agree...

[ErnstKompressor]

That is kinda what I was aiming at...Something like SPEC2000, but free/open source(a good cross-platform capable suite) or like XBench [xbench.com] for OS X(already free, but not cross platfrom). Both types of testing progs seem to give well-rounded complex test results.

I guess the meat of the question regards the dispute over using different compilers in relation to comparative testing...With the recent G5 debate focusing on the fairly noted use of GCC on both platforms instead of an intel-specific compiler, leadin

Benchmarking 101

[GlassHeart]

how should one go about testing systems to obtain the fairest results?

All benchmarks are fair unless you deliberately picked the criteria to favor one candidate over the other, or overgeneralize the collected results beyond reason. The bigger question is whether they are representative or real performance in real application.

What compilers are evenly matched amongst different hardware?

Compilers are not evenly matched among different source code, nevermind different hardware. I don't see how you can

You need to define your goal more clearly.

[Bob Bitchen]

What is it that you want to be able to say when you're done testing? And who do you want to say it to? By defining some of this at the start the pieces will more readily fall into place for you. There are a lot of tests suites out there now already and you should be able to find them once you know what you want to measure and what you want to report. This might be a good place for you to start: http://opensourcetesting.org/performance.php

In the Linux Kernel...

[Sepper]

Bogomips?
http://www.tldp.org/HOWTO/BogoMips.html

Not so easy...

[tetabiate]

It is hard to find a set of applications capable of testing every aspect of computer performance. Starting from scratch it demands a lot of time (and money) to make the applications portable enough to run on different architectures and operating environments (including compilers, filesystems, etc.) Then a database containing the results must be mantained for future reference. Only in these conditions the benchmark programs can be considered standard, it is perfectly justified to sell it for money. In conclu