[ previous ] [ next ] [ threads ]
 From:  Jim Thompson <jim at netgate dot com>
 To:  Braden McGrath <braden at mcmail dot homeip dot net>
 Cc:  m0n0wall at lists dot m0n0 dot ch
 Subject:  Re: [m0n0wall] OpenBSD Packet Filter in m0n0wall?
 Date:  Wed, 30 Mar 2005 12:27:33 -1000
Braden McGrath wrote:

>Jim Thompson [jim at netgate dot com] wrote:
>>Last I checked, openbsd suffered in terms of performance 
>>compared to FreeBSD and NetBSD.
>How did you check?  Maybe for some CPU-intensive apps it might run
>slower, but pf is actually quite quick, and OBSD in general runs very
>well on older hardware with minimal requirements.
>If there are any performance hits, they come as a tradeoff with
>SECURITY.  For instance, OpenBSD's entropy pool is much more
>sophisticated than any other *nix OS out there, with the exception of
>Linux using various security patches (that have replicated the entropy
>code from Open).  OpenBSD had randomly generated TCP sequence numbers
>before anyone else really knew why they mattered.  There are a whole lot
>of things that Open does "the right way" from a security point of
>view...  the downside is that if you want to do any *close* work on the
>project, you have to deal with Theo.  Mr. de Raadt is a subject of many
>rants on many lists across the web, Google can tell you that and I'm not
>about to duplicate the work here.  ;)

Aside from the issues (dealing with Theo de Raadt) that Brian points 
out, (reason enough to not touch OpenBSD with a wooden pole), OpenBSD 
advocates like to point out that their OS was secure *first*, but hardly 
ever admit that other platforms (FreeBSD, Linux, NetBSD) are now *as* 
secure, and a whole lot more stable. Having a firewall (or other 
security device) that craps out all the time isn't really any better 
than having one that has bugs. (Yes, there is a trade-off here.)

Further, OpenBSD only supports Atheros 802.11 cards via their 
"reverse-engineered" (HARDLY!) HAL, which is entirely incomplete if you 
understand the Atheros chipsets to any level of depth. The NetBSD 
net80211 layer is badly out of date, but at least the commiter on NetBSD 
is, um, committed to working on it.

Third, this is a "pretty interesting" read on FreeBSD .vs OpenBSD (.vs 
Linux 2.x and NetBSD):



The clear winner in the graph is Linux 2.6. OpenBSD does not scale at 
all, and even panics under high load.
NetBSD scales O(n), which is respectable for the grandfather of all the 
BSDs, but it is not a winning
performance. Linux 2.4 shows that there is work to be done; I give it 
the third place. FreeBSD looks like it
would scale O(1) if I could create more processes with it, but as long 
as I can't confirm it, I can only give it the
second place.


Whoa! Obviously, something is seriously broken in the OpenBSD memory 
management. OpenBSD is so
incredibly slow that compared to this performance, NetBSD looks like 
Warp 9, and Linux is not even on the
same chart.

Conclusion: Linux 2.6 is the clear winner, scaling O(1) in every 
respect. The clear loser is OpenBSD; I have
never seen bad performance of this magnitude. Even Windows would 
probably outperform OpenBSD.


I omitted the graphcs for Linux and FreeBSD because they were O(1), as 
expected. As you can see, it was
OpenBSD that showed the O(n) graph, and NetBSD that has the O(1) graph 
here. I am as surprised as you.
Believe me, I double and triple checked that gatling used kqueue on 
OpenBSD and that I hadn't switched the
results or graphs somehow.

The clear loser is, again, OpenBSD. Don't use OpenBSD for network 
servers. NetBSD appears to have found
some clever hack to short-circuit poll if there only are events for one 
of the first descriptors in the array.



Linux 2.6 scales O(1) in all benchmarks. Words fail me on how impressive 
this is. If you are using Linux 2.4
right now, switch to Linux 2.6 now!

FreeBSD 5.1 has very impressive performance and scalability. I foolishly 
assumed all BSDs to play in the same
league performance-wise, because they all share a lot of code and can 
incorporate each other's code freely. I
was wrong. FreeBSD has by far the best performance of the BSDs and it 
comes close to Linux 2.6. If you run
another BSD on x86, you should switch to FreeBSD!

Linux 2.4 is not too bad, but it scales badly for mmap and fork.

NetBSD 1.6.1 was treated unfairly by me because I only tested the stable 
version, not the unstable source
tree. I originally only wanted to benchmark stable versions, but 
deviated with OpenBSD and then with
FreeBSD. I should have upgraded NetBSD then, too. Nonetheless, NetBSD 
feels snappy, performs well overall,
although it needs work in the scalability department, judging from the 
old version I was using. Please note
that NetBSD was the only BSD that never crashed or panicked on me, so it 
gets favourable treatment for that.

OpenBSD 3.4 was a real stinker in these tests. The installation routine 
sucks, the disk performance sucks, the
kernel was unstable, and in the network scalability department it was 
even outperformed by it's father,
NetBSD. OpenBSD also gets points deducted for the sabotage they did to 
their IPv6 stack. If you are using
OpenBSD, you should move away now.

Finally, here is another performance comparison (this time just for 
PPPOE) that shows off OpenBSD's sluggishness. 

Now yes, some of these are based on old(er) versions of the various *BSD 
suites. Still, it shows that performance and stability were second to 
OpenBSD's "secure by default" mantra. I think performance and
stability are paramount in an embedded device.

Given what I've seen recently, NetBSD 2.x is *very* close to FreeBSD in 
terms of network performance. Frankly, the only reason to bother with 
NetBSD is to get something very m0n0-ish on non-x86 hardware. (Its 
easier than porting FreeBSD to xscale or mips.)

As for development models:

Linux is Kaustkian socialist, FreeBSD is Trotskyist, OpenBSD is 
Leninist, NetBSD is Maoist.