[ previous ] [ next ] [ threads ]
 From:  Jim Thompson <jim at netgate dot com>
 To:  m0n0wall dash dev at lists dot m0n0 dot ch
 Cc:  Fred Wright <fw at well dot com>
 Subject:  Re: [m0n0wall-dev] Re: [m0n0wall] The future
 Date:  Thu, 27 Oct 2005 22:04:46 -1000
I wasn't going to respond until Chris did. Some of what Fred says is 
just plain wrong.  Unlike Chris, I'm not afraid of having my head handed 
to me, thought I doubt it will happen in this forum.

Please forgive what seems like a "lashout" below.  Its not intended as 
same, but some of Fred's statements of "truth" are really unfounded 

Fred Wright wrote:

> Here are my thoughts:
>System Specs:
>Embedded platforms (e.g., Soekris & WRAP) should remain the primary target,
>regardless of what surveys say.  Even if lots of people want to run
>m0n0wall on boxes where the CPU lifetime after a fan failure is measured in
>milliseconds, those who are astute enough to use more reliable hardware
>without moving parts shouldn't be neglected, any more than Mac, Linux, and
>BSD desktop users should be neglected just because 95% of the desktops run
I'd advocate adding those VIA CPU platforms which run fanless for 
extended periods.  Even the dual 1GHz CPU "VIA DP-310" board (with 1 x 
10/100/1000 and 2 x 10/100 Ethernet) can run fanless.  Neither is VIA 
the only manufacturer of boards with their chips on them.   Picking up 
support for using Padlock to accelerate IPSEC would be a very good thing.

Should m0n0wall ever move to netbsd or openbsd (I'm not avocating 
either), several other CPU architectures could support the 'fanless' 

>The "fanless" requirement imposes a serious upper limit on CPU power which
>is *not* significantly alleviated by "Moore's law" evolution (especially
>with the x86 architecture).  Thus it's not reasonable to allow slothful
>code and assume that the performance problems can be covered up by throwing
>faster hardware at it.  And of course, something which runs well on slower
>hardware will run really well on faster hardware, while something that runs
>adequately on faster hardware may be unusable on slower hardware.
In general this is true, but it also possible that something which limps 
on 486 class processors (such as the Elan)
might *fly* on PII and up.   DES and AES are fine examples.   For 
example, Rijndael (the algorithm in AES) runs at 330KBps on a 50MHz 
486DX2, but 4.29MBps on a P90, and 12.88MBps on a 200MHz PII.   (128 
bit  keys/128 bit messages.)

You can consider an Elan 520 to be (effectivly) a 486 DX4 @ 100-133Mhz.

More telling is the "clock cycles per block"
486DX2 @ 50MHz  2370 cycles/block
P90: 320 cycles/block
PII @ 200MHz:  237 cycles/block

>Today, CF is available (at reasonable cost) in sizes pretty much as big as
>one would conceivably need for this sort of application, while RAM on the
>embedded platforms remains fixed and limited.  
I'd take it one step farther.   Try (just try) buying CF cards smaller 
than 128MB in any volume (we buy 200-500 at a time).

Thus, I think the "m0n0 is 8MB (or less)" limit needs to be relieved some.

>After a successful update, the roles get switched.  There are already some other devices that use
this approach.
I have code to do this (based on nanoBSD) if someone wants it.

>Although "object-oriented" seems to be many people's favorite bandwagon at
>the moment, the baggage that OO languages bring along isn't really
>appropriate for something like a firewall.
For the languages that most people use (C++, Perl, etc) OO is just 
syntactic sugar.

>In a security-conscious application, you really don't want a bunch of complex, obtuse, and often
buggy mechanisms hiding behind the language definition.  And when one uses
>an OO language, there's a tendency to frame everything in OO terms, whether
>there's a real benefit or not, due to the "when you're a hammer, everything
>look like a nail" principle.  Programmers who can't blow their noses
>without a box of "object-oriented" tissues shouldn't be doing firewall
Sticking with low-horsepower boxes and avocating OO is  a lot like 
shooting yourself in the foot.  Anyone looking at the runtime 
requirements for C++ or Perl with an embedded system as a target should 
be continually fighting the gag reflex.

>PHP has serious performance issues due to being an interpreted language,
>but that can be tolerated in the WebGUI, particularly if more attention
>were paid to performance issues.  I'd certainly get away from using it for
>non-GUI purposes.
Its OK as a 'runs-at-boot' configuration interpreter, but yeah, for 
anything that is CPU sensitive, its the wrong solution.
However, we do want to minimize the number of language runtimes availabe 
on the platform.

>1) Polling.  The purpose of polling is to avoid wasting CPU time in
>interrupt servicing, in order to make more CPU available to userland code.
>It does *not* have a significant effect on routing throughput (unless the
>NIC driver is pretty brain-damaged), since the number of packets handled
>per interrupt tends to self-regulate to the point of "keeping up". 
The opposite has been found in projects such as Click and Xorp.  (I can 
point to other research as well.)

On FreeBSD 4.11 Polling increased throughput 4X.  (On Linux polling 
increased throughput by an order of magnitude.)

>Because of this, it's not possible to determine maximum throughput merely by
>extrapolating CPU usage from lighter loads.  Thus, on a router, polling
>only helps ancillary functions (e.g.  the WebGUI) and userland-routed
>protocols like OpenVPN.  Meanwhile, polling-induced latency is much more
>significant in a router than in a host, for a number of reasons.
Please see my comment above, and review the XORP/Click papers (some of 
which really avocate FreeBSD, btw).

>2) Tuning.  Recently someone posted an excerpt from the FreeBSD tuning
>guide related to mbuf sizing.  But the stuff about mbufs needed per
>connection was all based on sizing *socket buffers*, not forwarded
>packets.  In fact, in a router, once you have enough buffering to avoid
>starvation, additional buffering actually *worsens* performance by
>increasing latency (due to longer queue lengths).
Only true if you allow queue lengths to grow unbouned.  There are times 
during a flow where the extra buffering will help, rather than harm 

>3) Network performance.  Practically all references to "network
>performance" (including benchmarks) are oriented to *server* performance,
>not *router* performance.  And claims like "fastest TCP/IP stack" are
>pretty meaningless when most of the "stack" isn't even involved in routing.
>On OS Choice:
>First of all, it should be noted that *no* *nix system is really ideal for
>routing purposes, for two main reasons:
>1) The networking code in *nix systems is designed primarily for the host
>role, with routing added as an afterthought.  The code is designed
>primarily for originating and receiving packets.  A router OS should really
>make packet *forwarding* the fundamental activity, with the local system
>simply being one of the "interfaces".
Which is why projects such as XORP and Click exist.

>2) Because *nix context switching is so "heavyweight", it's necessary to do
>a lot of things at interrupt level just to get decent performance.  In a
>well-designed microkernel system, context switching is fast enough to do
>practically everything in userland with good performance.  This improves
>robustness, debuggability, and CPU allocation.
Please provide examples that support this assertion.

>	FreeBSD:	Servers
>	NetBSD:		Portability
>	OpenBSD:	Security and Reliability
>	Dragonfly	Desktops
>Note that *none* of these actually set out to be optimized as a router OS.
>However, OpenBSD's emphasis on security and reliability made it an
>attractive choice for firewall/router applications, which led to greater
>developer attention to router issues, which led to increased use as a
>router, etc.  etc.  Thus, it's become the de facto BSD of choice for router
>applications, which receive a lot of developer attention, including the
>creation of things like pf and carp.  In general, most *nix-based routers
>use either Linux or OpenBSD, with the former choice being more due to the
>"bandwagon factor" than real benefit.
Juniper runs FreeBSD.  True this is the control plane, not the 
forwarding plane, but where are OpenBSD's "security" bennefits?

XORP primarily targets FreeBSD (4.11)

OpenBSD's team is far too willing to violate RFCs in their persuit to 
"rule the world" in my (not so) humble opinion.  Have you looked at the 
'ntp' implementation they ship?

Have you *looked* at the mess they made of 802.11 support in OpenBSD?

>The effect of FreeBSD's "focus" is quite apparent in the 5.x fiasco.
>FreeBSD's main goal is to provide the best-performing server OS on x86
>hardware, and it's done an excellent job of that (though recent Linux
>versions are giving it a run for its money).  But in some cases,
>server-oriented improvements inflict collateral damage on the routing code,
>as in 5.x.  
Above you say they're not connected, now you say they are.  

>There's no reason to suppose that this was a one-time thing -
>the FreeBSD developers are always going to pay more attention to server
>issues and router issues, because *not* doing so would distract them
>from server improvements.  
Nor can we say that OpenBSD will continue to be able to claim itself as 
the "most secure".  Past performance is no indicator of future trends, 
and all that.

>NetBSD would provide the largest choice of hardware platforms, but it's not
>clear that there are any platforms significantly interesting as routers
>that are supported by NetBSD and not OpenBSD.  Sure, there may be a few
>people wanting to run m0n0wall on their dual-NIC toasters, but that doesn't
>seem like a very good justification.  NetBSD is claimed to offer
>performance somewhere between FreeBSD and OpenBSD, but those are
>host-related claims.  And at least in V1.5.3, I found NetBSD to have the
>worst installer and the most "pickiness" about the hardware configuration
>of the three (never bothered trying Dragonfly).
But the 'installer' is a *host issue*, no?

>Thus, I consider OpenBSD to be the best choice, not primarily due to
>"better security" (although that's a factor), but because it's the one
>branch of BSD where the developers take routing applications seriously.
>The two main arguments that have been advanced against OpenBSD are:
>1) Performance.  Some have made, and others have parroted, claims that
>OpenBSD has substantially worse "networking performance" than FreeBSD.  But
>AFAIK these are all based on *server* performance.  The only *routing*
>performance comparison I've seen mentioned here was what Manuel did before
>starting m0n0wall, which was of course several versions ago for both
>systems.  I've not seen any information on routing performance comparisons
>between *current* versions of OpenBSD and FreeBSD.  Plus, due to the "focus
>factor", one would generally expect that comparison to shift more toward
>OpenBSD as the two systems evolve.
Just as OpenBSD's current advantages will tend to converge into FreeBSD.

>2) Atheros support.  While the concept of a vendor-supplied HAL makes
>sense, Atheros really screwed it up by providing it in binary-only form
>(for a pretty lame reason AIUI).  If they had done it right - i.e.
>providing the HAL as CPU- and OS-independent C source, then there would
>have been no issue. 
I'm certain (as in, I've asked and gotten answers) that they would have 
prefered this path as well.  It wasn't an option, due to the very simple 
fact that it is *trivial* to tune the Atheros chipset *way* out of the 
ISM (and U-NII) bands.   This, in effect means that an "open HAL" would 
(not could, *would*) result in substantial interference issues with 
users of licensed spectrum.   While I could detail the various license 
holders who could scream (and sue) over this, I will point out one very 
vocal set of "licensed" users who would pull every trick in the book the 
second an "open HAL" resulted in detectible interference:   HAM Radio 
Operators (especially the ARRL).  You see, the (nearly world-wide) 13cm 
band runs from 2304MHz to 2450Mhz.

> As it is, some people grudgingly accepted the
>"official" HAL, while someone else developed an open-source replacement HAL
>by reverse engineering Atheros's, and that was introduced in the latest
>version of OpenBSD.  
And it... doesn't work.  Its total crap.  Have you tried it? Moreover 
its clear (for those of us who can look at both the openbsd and "closed" 
HAL source code that there is clear copyright infringement in the 
(supposed) "reverse engineered" HAL in OpenBSD.

Want to get (your customer's) sued?  Run the "reverse engineered" HAL.

>You'll find things of the form "It's evil because it's
>based on reverse engineering" and "Is this really legal", as well as "When
>is OpenBSD's open-source HAL going to be ported to our platform so we can
>junk the binary-only crock?".  I have yet to see any actual behavioral
>comparisons between the two. 
I have.  I've run them.  We *actively refuse* to support people running 
the "open HAL" code on products we sell.

> Of course the best scenario would be if the open-source HAL became sufficiently popular to shame
Atheros into providing the "official" HAL in source form.  :-)
Its not going to happen.   First it would have to work, then the authors 
of the "open source HAL" would end up in court, having to defend 
themselves from a  copyright infringement suit.  Then they would end up 
in court having to defend themselves from the wrath of "spectrum 
licensees" and the regulatory agencies that granted (or sold) those 

>Note that if there really were a significant advantage in using the binary
>HAL in OpenBSD, it could be used with a "wrapper" in the same manner as it
>is in FreeBSD.  
Too bad OpenBSD screwed with net80211, or one could merely "replace" one 
with the other.

>One could even do a side-by-side comparison of two builds
>of OpenBSD, one with the binary HAL and one with the open-source HAL.  But
>I wouldn't go to that much trouble without determining that the effort is
I've run OpenBSD and FreeBSD on two CF cards on the same hardware 
(soekris and wrap boards) with the same radio cards.   OpenBSD has a 
hard time even maintaing associations.

>More generally, it's important to make a distinction between what devices
>are "officially" supported by OpenBSD, and what devices are supported by
>third-party drivers.  OpenBSD is more rigorous about the definition of
>"open-source" than some other systems, and hence can't officially include
>drivers that aren't fully open-source.  I'd expect the same to be true of
>Linux due to GPL requirements. 
Non-GPL drivers will get more difficult to incorporate into linux over 
time.   this is especially true as the strings are
tightened on object-only drivers that have to semi-obviously make 
reference (and incorporate) GPL liccensed header files, data strcutures, 

A carefull analysis of the Atheros-supplied HAL will show an elegant 
method for avoiding that mess.

But I digress.

> While I don't consider an open-source
>system to be "tainted" by the inclusion of binary *firmware* to be loaded
>into peripherals, if any code executed by the CPU isn't open-source, then
>the system can't be honestly described as open-source without a suitable
>disclaimer.  And one has to question the wisdom of including "black-box"
>code in a firewall.
>It's also worth noting that binary driver code is at the very least
>CPU-specific, making it a major hassle for an OS that supports multiple CPU
>architectures.  FreeBSD tolerates this more easily due to being, for all
>practical purposes, x86-only, but other OSes have more difficulty with it.
>And there seems to be quite a bit of interest in running m0n0wall on
>non-x86 router hardware.
The Atheros HAL object is identical (even in binary form) between linux 
on x86, netbsd on x86 and freebsd on x86.  There is an OS-dependent 
'wrapper' for it.

For netbsd, mach (as used in Apple's product) and linux the same is true 
for PowerPC (albeit with big/little endian being different 
architectures. 64/32 bit would be an issue if NetBSD and linux didn't 
both run the kernel in 32-bit mode.  (Mach too.))

Reducing to NetBSD and Linux, this is still true for
ARM9 (big and little endian)
MIPS big and little endian

Your point (as far as it relates to the "closed source" (Atheros 
supplied) HAL is moot.

>OpenBSD and NetBSD are sufficiently similar internally that drivers (and
>even in some cases kernel patches) can usually be ported from one to the
>other fairly easily.  Thus, any device supported by NetBSD can be at least
>unofficially supported by OpenBSD without too much difficulty.  FreeBSD
>is sort of the "odd man out" in this department.
This is mostly because Theo "forked" OpenBSD after the NetBSD guys 
kicked him out for being ... an asshole.
Only *then* did we start to get "security" as a focus.

>The bottom line is that the two major arguments against OpenBSD are at the
>very least lacking in data, if not simply wrong.
Try again.  Please.

>Issues with OpenBSD:
>Here are some issues I'm aware of that would need to be addressed in moving
>to OpenBSD:
>1) FTP NAT performance:  Unlike IPFilter's in-kernel FTP proxy, pf relies
>on the userland ftp-proxy program.  As far as the control connection goes,
>this is a big win, since handling the control connection at the packet
>level is a horribly kludgy and fragile mess.  So much so that Cisco has
>fixed bugs in that area within the past year.  And control-connection
>performance is a non-issue.  However, ftp-proxy also proxies the data
>connection through userland, which involves a significant performance
>penalty.  It can avoid that in two out of the four combinations of
>client/server and active/passive, but not the remaining two.  There's an
>(unofficial) alternative called "ftpsesame", which manipulates the filter
>tables to forward the data connection properly, but it has a horrible
>packet-oriented bpf-based kludge for the control connection.  Ideally, one
>would like a hybrid of the two approaches, but AFAIK none such exists ATM.
>Note that this is not strictly an OpenBSD/FreeBSD issue, but rather a
>pf/IPFilter issue, so pfsense is presumably already living with the extra
>overhead, and since practically everyone seems to agree that it makes sense
>to move to pf instead of IPFilter, it's an issue regardless of OS choice.
>Fixing it might be a bit easier in OpenBSD, simply because it's the "native
>habitat" of both pf and ftp-proxy, without "compatibility layers" getting
>in the way.
This is just arrogant.  pf is fully supported in FreeBSD 6 (and above?).

>Advantages of OpenBSD:
>1) Strong orientation toward router-related issues.
>2) Proactive security.  Some have described this as a "solved problem",
>but it doesn't remain "solved" as long as the code keeps changing.
But as this code chances, the other two "BSDs" tend to pick up the 
changes, "quickly". 

Not all security issues are addressed first in OpenBSD.

>3) Most aggressive development in cryptographic areas, including support
>for hardware accelerators.
Nevermind that Sam Leffler is the primary author of many of these, and 
he develops them for FreeBSD first.

>4) Widely desired features like pf and CARP in their "native habitat".
Now a non-issue.

>5) Ability to run on non-x86 hardware.
>6) IPv6 support that's actually usable.  At least as recently as 5.3,
>FreeBSD had conflicts between IPv6 and FAST_IPSEC which could cause kernel
>panics.  Although IPv6 isn't exactly taking the world by storm, it
>definitely needs to be in any reasonable plan for the future.
Could we discuss 6.0 or better, please.  Especially given that neither 
m0n0wall or pfSense are likely to be based on any 5.x release of FreeBSD?

>7) Improved IPsec support, including NAT-T.
>8) Development cycle based on conservative goals, where new releases have
>a high probability of being adoptable without significant problems.
Anyone who has done software development for as long as either of us 
knows that this kind of statement can't stand for long.