Benchmarking Unix Operating Systems

FreeBSD 6.1, GNU/Linux 2.6.16, NetBSD 3.0 and OpenBSD 3.9

Table of contents [hide]

Abstract

Before we start with this document and the tests involved, I would like to say that all the tests have been made under the same conditions and performed several times to ensure that no random success could alter the results.

Apart from this, these tests have been performed just for fun, because I like to compare BSD systems to GNU/Linux systems. If you can't hold your feelings (either hurt or strengthened) provoked by these tests, then please contact me.

Introduction

This benchmark is something that I really wanted to do since some time ago. I've always been doubting between using BSD systems or rather GNU/Linux systems, and the truth is that I've never been sure. Currently I use Debian GNU/Linux Sid, but I admit that I've always been in love with NetBSD.

Some of the most astonishing surprises that I have found on the way have been the following ones:

The tests

I split the benchmark tests in four parts:

In the subjective tests I manually measured the load time when using big programs like Gnome (2.14.1), KDE (3.5.2) or Firefox (1.5.0.3). I also measured the time that takes Vim to open huge files, the CPU usage with 1, 2 and 4 instances of XMMS and the CPU usage when playing an AVI video with Totem, using the libxine.

In some of the technical tests, we find unixbench (4.1.0), and some benchmarks from people from NetBSD (see Bibliography).

The system settings

Linux was a GNU/Linux Gentoo 2006.0 with a Linux 2.6.16 kernel and some of the kernel options were: PREEMPT, ACPI, AGP and CFQ IO.

FreeBSD was a 6.1 RELEASE, with SCHED_44BSD, PREEMPT, ACPI, APIC, VFS_AIO, AGP and COMPAT_LINUX.

NOTE: I know that SCHED_ULE is newer and better when using multi-threading, but I've also read that it performs worse on uniprocessor machines, so I decided to use the classic 4.4BSD scheduler.

NetBSD 3.0 is another candidate. The kernel used the options: USER_LDT, QBUF_READPRIO, UFS_DIRHASH, ACPI and AGP. Unfortunately, there's no Bluetooth nor DRI support (maybe in the 4.0).

Finally, OpenBSD 3.9, in which I made some changes like activate DUMMY_NOPS (a speed-up hack), APERTURE (necessary for Xorg) and the VM86 option.

The hardware

Test 1: Subjective tests

NOTE: I wanted to compare also Debian for personal reasons, since I use Debian GNU/Linux as common system on my laptop. All the software in Debian GNU/Linux is built with GCC 4 for i386 and, consequently, it is not optimized.

NOTE: The values indicated with a '-' mean that the test, in that case, couldn't be performed due to some limitation on the system.

Test FreeBSD Gentoo Debian NetBSD OpenBSD
Start Firefox 11.6s 8.0s 8.5s 12.0s 14.4s
Start Gnome 19.6s 13.9s 19.9s 26.7s 29.8s
Start KDE 30.2s 18.0s 22.0s 17.6s 25.7s
Open 470Mb with Vim 33.5s 27.6s 23.4s 36.7s 27.3s
CPU xine (AVI video) 14.5% 15.0% 18.7% 10.6% 26.0%
CPU XMMS (1/2/4 procs) 0.0/0.0/- (1) 0.0/0.3/0.7 0.0/0.3/0.6 0.0/-/- (2) (3)

TEST WINNER: Both Gentoo and Debian GNU/Linux.
TEST LOSER: OpenBSD so far.

(1) In FreeBSD, even though I increased the audio channel number from 4 to 8, I wasn't able to play more than three files at the same time.

(2) NetBSD doesn't support yet multiple audio channels, but there's a lot of work done in this field for later implementations.

(3) It is unbelievable that being on the year 2006, we still need to investigate to play an MP3 file with XMMS. I didn't have the time nor the gats to start doing such a research, so I left the test.

Test 2: VFS tests

NOTE: The test concerning to the filesystem depend on the default filesystem on every distribution. These are the filesystems which were use and their options:

Test FreeBSD 6.1 Linux 2.6.16 NetBSD 3.0 OpenBSD 3.9
tar jxvf 45s 1m29s 33.9s 44.8s
tar jcvf 2m55s 2m57s 2m30s 2m32s
rm -fr dir/ 7.7s 9.7s 5.5s 8.3s
Bonnie++ test (FS) 4m08s 5m01s 4m57s 8m10s
memory read 544.7MB/s 544.7MB/s 914.3MB/s 533.3MB/s
memory write 393.8MB/s 387.9MB/s 581.8MB/s 400.0MB/s

TEST WINNER: NetBSD 3.0 with UFS2, so far.
TEST LOSER: GNU/Linux with EXT3.

Test 3: Performance tests

NOTE: The values without unit are reference points given by unixbench, the greater they are, the better the system performance.

Test FreeBSD 6.1 Linux 2.6.16 NetBSD 3.0 OpenBSD 3.9
Convert JPEG->PNG 2m32s 2m20s 2m31s 2m46s
Convert AVI->MPEG 7m59s 7m51s 9m57s 8m16s
syscall overhead 477.6 729.8 596.7 517.5
pipe throughoutput 725.5 607.6 706.1 735.5
pipe-based context switch 500.1 56.3 553.6 739.0
process creation 712.1 847.7 721.3 250.5
execl throughoutput 331.3 772.4 562.9 263.3

TEST WINNER: Linux 2.6 clearly, although note the surprise that OpenBSD 3.9 outperforms NetBSD and FreeBSD, fact that didn't happen one year ago.
TEST LOSER: Both FreeBSD and NetBSD almost at the same level.

Test 4: Latencia tests

NOTE: The latency value is a mean value from several tests performed. The lower the value, the faster the response of the system.

NOTE: The results are expressed with the following shorts: us (microseconds) y ns (nanoseconds).

Test FreeBSD 6.1 Linux 2.6.16 NetBSD 3.0 OpenBSD 3.9
context latency 1559ns 1024ns 1289ns 733ns
exec latency 360us 342us 71us 205us
fork latency 79us 82us 35us 92us
signal latency 1385ns 752ns 774ns 580ns
syscall time 234ns (*) 21ns 184ns (*) 190ns (*)

TEST WINNER: NetBSD is the winner, but OpenBSD is really close and also GNU/Linux.
TEST LOSER: FreeBSD is the clear loser.

(*) This great difference is because BSD systems don't implement the sysenter/sysexist fast system call methods for the i386 architecture.

Conclusions

Here I'd like to separate the analysis and, by the way, widen it with some facts that we can't see just from the results above. Starting with the installation, FreeBSD is the easiest system to install, followed in order by NetBSD, Debian GNU/Linux and OpenBSD. Gentoo is too difficult to install if we have in mind that many GNU/Linux distributions offer an user-friendly and totally automated installation process.

Apart from this, the easiest system to use from the beginning is also FreeBSD. For Gentoo, Debian GNU/Linux, NetBSD and OpenBSD, I had to read about an hour to start using them (compared with the half an hour I spent on learning the basis of FreeBSD). pkgsrc, in my opinion, was the best I tried, but I also reckon that Debian GNU/Linux's apt is really great (and it's faster than NetBSD's pkgsrc).

As you can see in the VFS tests, NetBSD has improved so much and, indeed, it is the one that shows the best results at almost every level, surpassing GNU/Linux in most of the cases. Also is true that the NetBSD's pagedaemon is not good enough when working under heavy overloads, whereas FreeBSD's one is much better in these cases, as you can notice from the bonnie++ test. As a last note, UFS2 with NetBSD turned out to be the most efficient filesystem in almost every test, but not when I/O are very heavy, where the VFS becomes really slow. FreeBSD also demonstrated that it has done a great job with UFS2. Against UFS2, EXT3 is the only one that supports journaling and, therefore, it is the fastest when recovering.

Finally, according to the results, we can deduce that GNU/Linux is a general winner, but NetBSD offers a great performance, near the Linux level and sometimes even surpassing it. OpenBSD turned out to be a great system, at least when talking about performance and responsiveness, getting close to NetBSD and surpassing my expectations. Once again, FreeBSD has been the worst of the candidates, showing that its desktop features has margined the core system performance.

I'd like that some of the readers who have reached till here to take a look at some benchmarks performed three years ago and let them build up their own conclusions according to the evolution of every system.

In those benchmarks, the general result was like: Linux 2.6, Linux 2.4, FreeBSD 5.1, NetBSD 1.6.1 and then OpenBSD 3.4. Nowadays, and according to my tests, the general result would be like: Linux 2.6, NetBSD 3.0, OpenBSD 3.9 and then FreeBSD 6.1.

Bibliography

Scalability Benchmarks
Benchmark Comparision of NetBSD 2.0 and FreeBSD 5.3
Tuning NetBSD VM behaviour (swap usage)
Gentoo Handbook (2006.0 edition)
Gnome 2.14.1, KDE 3.5.2, Firefox 1.5.0.3, XMMS 1.2.10 and Vim 6.4
QEMU 0.8.1 (no in-kernel accel), used to take screenshots
ImageMagick 6.2.4.5, used for JPEG/PNG transformation tests
ffmpeg 0.4.9-CVS, used for video transformation tests
bonnie++ 1.93, used for the filesystem tests
totem 1.4, used for the video playing tests
unixbench 4.1.0, used for the performance tests
NetBSD's gmcgarry's benchmarks

www.claudiocamacho.org
Updated on Saturday, 13 September 2008 12:37