[ previous ] [ next ] [ threads ]
 From:  Chris Buechler <cbuechler at gmail dot com>
 Cc:  m0n0wall dash dev at lists dot m0n0 dot ch
 Subject:  Re: [m0n0wall-dev] Re: [m0n0wall] The future
 Date:  Thu, 27 Oct 2005 23:38:49 -0400
I'll dare respond to a guy that knows more *nix than anybody that's
ever popped their head up on this list, and has been working on *nix
about as long as I've been alive.

/me waits to get owned  :)

I guess I have as many questions as anything though.

On 10/27/05, Fred Wright <fw at well dot com> wrote:
> 1) Polling.  The purpose of polling is to avoid wasting CPU time in
> interrupt servicing, in order to make more CPU available to userland code.

doesn't the same avoidance of CPU time wasted lead to increased
throughput though?  In most cases, your bottleneck (on CPU's not of
sufficient speed to achieve wire speed) is CPU time spent on
interrupts, i.e. the CPU gets pegged at 100%.  I've done a lot of
performance testing, but never with polling enabled.  Manuel has
stated in an email (to me personally) that he's tested with and
without polling and saw about a ~50% performance increase with polling
enabled on a 4501.

> 2) Tuning.  Recently someone posted an excerpt from the FreeBSD tuning
> guide related to mbuf sizing.  But the stuff about mbufs needed per
> connection was all based on sizing *socket buffers*, not forwarded
> packets.  In fact, in a router, once you have enough buffering to avoid
> starvation, additional buffering actually *worsens* performance by
> increasing latency (due to longer queue lengths).

I know this is primarily not relevant for firewalling purposes, but
does this not come into play with larger loads?  Seems every once in a
while somebody stops in with a large network (couple thousand+ hosts)
and is running out of mbufs.

"out of mbufs" seems to crop up for a few people periodically for no
apparent reason under light loads.  From some brief research I did a
while back, I believe I came to the rough guess that it was probably
driver bugs, or something of that nature, though I really have no

> 3) Network performance.  Practically all references to "network
> performance" (including benchmarks) are oriented to *server* performance,
> not *router* performance.

not any that I quoted.  Most I quoted I did myself, in a firewalling
scenario.  Some information taken from other benchmarks I've come
across in the past (like the Linux numbers).

>        Dragonfly       Desktops

desktops?  that's probably the last thing I'd consider to be the focus
of DFly.  If I had to sum it up in one word, it'd probably be
clustering.  A lot of Dillon's lofty goals are in that area.  That's
beside the point though, I completely agree that it isn't suitable.

> The effect of FreeBSD's "focus" is quite apparent in the 5.x fiasco.
> FreeBSD's main goal is to provide the best-performing server OS on x86
> hardware, and it's done an excellent job of that (though recent Linux
> versions are giving it a run for its money).  But in some cases,
> server-oriented improvements inflict collateral damage on the routing code,
> as in 5.x.  There's no reason to suppose that this was a one-time thing -
> the FreeBSD developers are always going to pay more attention to server
> issues and router issues, because *not* doing so would distract them
> from server improvements.  So even if 6.x completely repairs the damage
> caused by 5.x, there's no reason to suppose that the same thing won't
> happen again in 7.x or whatever.

this is a good point.

> After all, 5.x went through at least four
> versions wherein they either didn't notice or didn't care that they had
> seriously degraded routing performance.

they knew.  I had email discussions with Robert Watson re: the
performance problems we ran into in 5.3, and the general statement was
exactly as you said above - the kernel changes in 5.x provide
measurable performance benefits in many server scenarios, but really
hurt network performance, especially in firewalling scenarios on low
end hardware.  I wouldn't say they didn't care, they just had so many
things to work on going forward that it couldn't reasonably be fixed
in 5.x.  FreeBSD 6.0 is already reportedly as much as 30% or more
faster than 4.x in firewalling scenarios with SMP, but it still lags
quite a bit behind in uniprocessor environments.  With multiple core
processors becoming the norm going forward in all but the embedded
world, they're certainly moving in the right direction for the future
for the vast majority of the project.

> The two main arguments that have been advanced against OpenBSD are:
> 1) Performance.  Some have made, and others have parroted, claims that
> OpenBSD has substantially worse "networking performance" than FreeBSD.  But
> AFAIK these are all based on *server* performance.  The only *routing*
> performance comparison I've seen mentioned here was what Manuel did before
> starting m0n0wall, which was of course several versions ago for both
> systems.  I've not seen any information on routing performance comparisons
> between *current* versions of OpenBSD and FreeBSD.

That's not entirely true.  I have tested a recent OpenBSD and FreeBSD
6 in firewalling scenarios.  I didn't make note of the numbers, so I'm
going to put together some new tests with the latest of everything. 
This was the latest of both around June or July of this year.  On a
WRAP or 4801, OpenBSD measured at around 20 Mb, FreeBSD 6.0 at around
30 Mb, and FreeBSD 4.11 at around 45 Mb single TCP connection max
throughput.  OpenBSD and FreeBSD 4.11 don't seem to degrade much as
you increase pps, while FreeBSD 6.0 does see some degradation. 
(though not nearly as severe as 5.3, which could do about 11 Mb single
TCP stream max through a 4501, but only 3.5 Mb when hammered with
bittorrent traffic).

> 2) Atheros support.

I'd expand this to wireless support in general.  It's not only the
drivers that are lacking.

> While the concept of a vendor-supplied HAL makes
> sense, Atheros really screwed it up by providing it in binary-only form
> (for a pretty lame reason AIUI).  If they had done it right - i.e.
> providing the HAL as CPU- and OS-independent C source, then there would
> have been no issue.

I'm no wireless guru, but I believe Jim Thompson, who I would
certainly consider a wireless guru, has stated in the past something
along the lines of the Atheros drivers are binary because much of the
control of the hardware is done via the driver, as far as staying in
compliance with FCC and other regulations around the world.  Given the
ability to bypass this, people no doubt would.

> I have yet to see any actual behavioral
> comparisons between the two.

Jim Thompson, who I'd consider unbiased and more than qualified to do
this testing, stated OpenBSD's ath support just doesn't work. 

> One could even do a side-by-side comparison of two builds
> of OpenBSD, one with the binary HAL and one with the open-source HAL.  But
> I wouldn't go to that much trouble without determining that the effort is
> warranted.

I have no idea what's involved with that, but even if that kind of
effort was put into it, you're then in a situation where you have the
driver support, but the software support is very poor in comparison to
FreeBSD 6.0.  Again, I'm no wireless expert, but a *lot* of effort has
been put into the wireless support in 6.  It's my understanding that
it has many features that aren't in any other BSD, and it's on par
with Linux for the most part.

> And one has to question the wisdom of including "black-box"
> code in a firewall.

I can see that argument to some extent, but the computing world
revolves around black box code, in firewalls and the vast majority of
OS installations.  Even if all your OS source is open, the BIOS code
likely isn't.  There will likely always be "black box" code somewhere.

> The bottom line is that the two major arguments against OpenBSD are at the
> very least lacking in data, if not simply wrong.

Just lacking in data, they're correct.  whether them being correct is
a major issue is another question.

> 7) Improved IPsec support, including NAT-T.

FreeBSD 6 has this now too, with a NAT-T kernel patch that hasn't yet
been incorporated (but seems to work fine).

I'll give you the rest of the points.

After this post, OpenBSD is sounding a bit more attractive to me.

But there are some major outstanding issues if OpenBSD is to be
successfully implemented, like:
- PPTP server, passthrough, and WAN support
- implementing captive portal, traffic shaping, and firewalling with a
single firewalling package (pf) - the reason for the captive portal
change back to ipfw in pfsense was due to the fact that it was
extremely difficult to get the rules right for all three of those
simultaneously (with the primary issue being CP and ALTQ being very
difficult to get and keep working together properly)

This still depends on what our end goal is though.  If we want good
wireless support, FreeBSD is our best option by far.  If we don't care
to almost completely ditch wireless support, and if all the
outstanding issues can be resolved satisfactorily, OpenBSD sounds like
our best bet.

Wireless support seems pretty important though.  27% of our users
(that answered the survey) are currently using it, 63% ranked it as a
3 or higher in importance with only 20% saying not important, and 72%
that don't currently use wireless claim they would if it included
improved hardware and software support.  IMO, that's too big of a
number to choose OpenBSD and hence throw out wireless support.

There are also some major issues to wrapping a GUI around ALTQ.  I'm
trying to get some details on the shaping and CP issues encountered in
pfsense, and I'll post back if I can get some further info.