[ previous ] [ next ] [ threads ]
 From:  Paul Taylor <ptaylor at addressplus dot net>
 To:  telenieko at telenieko dot com
 Cc:  m0n0wall dash dev at lists dot m0n0 dot ch
 Subject:  Re: MFS
 Date:  Fri, 28 Oct 2005 22:57:35 -0400
Actually, I initially disliked the MFS approach...

But, once I gave it some thought, I see an advantage to it...  How  
likely is it that you can get hacked and have unknown software  
running on your machine across a reboot?  With MFS, it's pretty  
unlikely.  (Now, I'm not a big BSD or Linux guy, so I could be all  
wet here, but it seems unlikely to me that your base image could be  
easily penetrated.)  If your Monowall machine gets hacked and you  
want to ensure it's safe, unplug the WAN and reboot.  We all know  
that if a piece of malware takes hold on your Windows box, there is  
no telling how far it will embed itself in the system...  A simple  
reboot will do nothing in Windows to get rid of malware... Hmm...   
Perhaps an MFS version of Windows?.. :)  But seriously....

I don't know if security was the reason MFS was selected to start  
with, or simply to conserve space.. (It's good at that!)

But, once I got a development VM created using the instructions on  
the main monowall site, it wasn't bad at all.  I first added an SSH  
client so that I could test changes without having to load a new  
image and reboot.  With that in place, development really got moving...

I will admit that the built-your-own MFS approach may be more work  
than a lot of people are willing to go to get started...  But, I  
would be willing to say that if someone close to the project would  
create known-good development VM's and then wrote a bit of  
instruction just explaining the layout and how things work in  
general, I imagine that we'd have more active developers on the  

Of course, there is a LOT that I learned about FreeBSD by building my  
own, so perhaps that's a good step to require of people...  :)  On  
the other hand, having someone who is a GUI design wizard (but not  
necessarily a great app programmer) working with us would be a good  
thing, especially on some of the harder-to-grasp concepts like  
traffic shaping...


On Oct 28, 2005, at 5:04 AM, Marc wrote:

> Oh! there was one think we didn't care of....
> Currently if you want to touch anything in m0n0 you need a FreeBSD due
> to the MFS approach, most development of m0n0 once you have the
> contents of mfsroot is platform independent as you can boot it from
> pxe for testing.
> In my case for example I really hate changing anything of m0n0 'cause
> I have to boot a FreeBSD on a VM, the mount the mfs on it change it,
> umount, send it to my linux box and then do whatever I want with it.
> Are there any real possible alternatives to the MFS approach? or at
> least the UFS filesystem, that way the development of m0n0 would be
> much more platform independent what could attract more developers!
> cu,
> Marc.
> On 10/28/05, Jim Thompson <jim at netgate dot com> wrote:
>> I wasn't going to respond until Chris did. Some of what Fred says is
>> just plain wrong.  Unlike Chris, I'm not afraid of having my head  
>> handed
>> to me, thought I doubt it will happen in this forum.
>> Please forgive what seems like a "lashout" below.  Its not  
>> intended as
>> same, but some of Fred's statements of "truth" are really unfounded
>> opinions.
>> Fred Wright wrote:
>>> Here are my thoughts:
>>> System Specs:
>>> -------------
>>> Embedded platforms (e.g., Soekris & WRAP) should remain the  
>>> primary target,
>>> regardless of what surveys say.  Even if lots of people want to run
>>> m0n0wall on boxes where the CPU lifetime after a fan failure is  
>>> measured in
>>> milliseconds, those who are astute enough to use more reliable  
>>> hardware
>>> without moving parts shouldn't be neglected, any more than Mac,  
>>> Linux, and
>>> BSD desktop users should be neglected just because 95% of the  
>>> desktops run
>>> Windows.
>> I'd advocate adding those VIA CPU platforms which run fanless for
>> extended periods.  Even the dual 1GHz CPU "VIA DP-310" board (with  
>> 1 x
>> 10/100/1000 and 2 x 10/100 Ethernet) can run fanless.  Neither is VIA
>> the only manufacturer of boards with their chips on them.    
>> Picking up
>> support for using Padlock to accelerate IPSEC would be a very good  
>> thing.
>> Should m0n0wall ever move to netbsd or openbsd (I'm not avocating
>> either), several other CPU architectures could support the 'fanless'
>> requirement.
>>> The "fanless" requirement imposes a serious upper limit on CPU  
>>> power which
>>> is *not* significantly alleviated by "Moore's law" evolution  
>>> (especially
>>> with the x86 architecture).  Thus it's not reasonable to allow  
>>> slothful
>>> code and assume that the performance problems can be covered up  
>>> by throwing
>>> faster hardware at it.  And of course, something which runs well  
>>> on slower
>>> hardware will run really well on faster hardware, while something  
>>> that runs
>>> adequately on faster hardware may be unusable on slower hardware.
>> In general this is true, but it also possible that something which  
>> limps
>> on 486 class processors (such as the Elan)
>> might *fly* on PII and up.   DES and AES are fine examples.   For
>> example, Rijndael (the algorithm in AES) runs at 330KBps on a 50MHz
>> 486DX2, but 4.29MBps on a P90, and 12.88MBps on a 200MHz PII.   (128
>> bit  keys/128 bit messages.)
>> You can consider an Elan 520 to be (effectivly) a 486 DX4 @  
>> 100-133Mhz.
>> More telling is the "clock cycles per block"
>> 486DX2 @ 50MHz  2370 cycles/block
>> P90: 320 cycles/block
>> PII @ 200MHz:  237 cycles/block
>>> Today, CF is available (at reasonable cost) in sizes pretty much  
>>> as big as
>>> one would conceivably need for this sort of application, while  
>>> RAM on the
>>> embedded platforms remains fixed and limited.
>> I'd take it one step farther.   Try (just try) buying CF cards  
>> smaller
>> than 128MB in any volume (we buy 200-500 at a time).
>> Thus, I think the "m0n0 is 8MB (or less)" limit needs to be  
>> relieved some.
>>> After a successful update, the roles get switched.  There are  
>>> already some other devices that use this approach.
>> I have code to do this (based on nanoBSD) if someone wants it.
>>> Although "object-oriented" seems to be many people's favorite  
>>> bandwagon at
>>> the moment, the baggage that OO languages bring along isn't really
>>> appropriate for something like a firewall.
>> For the languages that most people use (C++, Perl, etc) OO is just
>> syntactic sugar.
>>> In a security-conscious application, you really don't want a  
>>> bunch of complex, obtuse, and often buggy mechanisms hiding  
>>> behind the language definition.  And when one uses
>>> an OO language, there's a tendency to frame everything in OO  
>>> terms, whether
>>> there's a real benefit or not, due to the "when you're a hammer,  
>>> everything
>>> look like a nail" principle.  Programmers who can't blow their noses
>>> without a box of "object-oriented" tissues shouldn't be doing  
>>> firewall
>>> programming.
>> Sticking with low-horsepower boxes and avocating OO is  a lot like
>> shooting yourself in the foot.  Anyone looking at the runtime
>> requirements for C++ or Perl with an embedded system as a target  
>> should
>> be continually fighting the gag reflex.
>>> PHP has serious performance issues due to being an interpreted  
>>> language,
>>> but that can be tolerated in the WebGUI, particularly if more  
>>> attention
>>> were paid to performance issues.  I'd certainly get away from  
>>> using it for
>>> non-GUI purposes.
>> Its OK as a 'runs-at-boot' configuration interpreter, but yeah, for
>> anything that is CPU sensitive, its the wrong solution.
>> However, we do want to minimize the number of language runtimes  
>> availabe
>> on the platform.
>>> 1) Polling.  The purpose of polling is to avoid wasting CPU time in
>>> interrupt servicing, in order to make more CPU available to  
>>> userland code.
>>> It does *not* have a significant effect on routing throughput  
>>> (unless the
>>> NIC driver is pretty brain-damaged), since the number of packets  
>>> handled
>>> per interrupt tends to self-regulate to the point of "keeping up".
>> The opposite has been found in projects such as Click and Xorp.   
>> (I can
>> point to other research as well.)
>> http://pdos.csail.mit.edu/papers/click:tocs00/paper.pdf
>> http://info.iet.unipi.it/~luigi/bsdcon01/polling/mgp00001.html
>> On FreeBSD 4.11 Polling increased throughput 4X.  (On Linux polling
>> increased throughput by an order of magnitude.)
>>> Because of this, it's not possible to determine maximum  
>>> throughput merely by
>>> extrapolating CPU usage from lighter loads.  Thus, on a router,  
>>> polling
>>> only helps ancillary functions (e.g.  the WebGUI) and userland- 
>>> routed
>>> protocols like OpenVPN.  Meanwhile, polling-induced latency is  
>>> much more
>>> significant in a router than in a host, for a number of reasons.
>> Please see my comment above, and review the XORP/Click papers  
>> (some of
>> which really avocate FreeBSD, btw).
>>> 2) Tuning.  Recently someone posted an excerpt from the FreeBSD  
>>> tuning
>>> guide related to mbuf sizing.  But the stuff about mbufs needed per
>>> connection was all based on sizing *socket buffers*, not forwarded
>>> packets.  In fact, in a router, once you have enough buffering to  
>>> avoid
>>> starvation, additional buffering actually *worsens* performance by
>>> increasing latency (due to longer queue lengths).
>> Only true if you allow queue lengths to grow unbouned.  There are  
>> times
>> during a flow where the extra buffering will help, rather than harm
>> throughput.
>>> 3) Network performance.  Practically all references to "network
>>> performance" (including benchmarks) are oriented to *server*  
>>> performance,
>>> not *router* performance.  And claims like "fastest TCP/IP stack"  
>>> are
>>> pretty meaningless when most of the "stack" isn't even involved  
>>> in routing.
>>> On OS Choice:
>>> -------------
>>> First of all, it should be noted that *no* *nix system is really  
>>> ideal for
>>> routing purposes, for two main reasons:
>>> 1) The networking code in *nix systems is designed primarily for  
>>> the host
>>> role, with routing added as an afterthought.  The code is designed
>>> primarily for originating and receiving packets.  A router OS  
>>> should really
>>> make packet *forwarding* the fundamental activity, with the local  
>>> system
>>> simply being one of the "interfaces".
>> Which is why projects such as XORP and Click exist.
>>> 2) Because *nix context switching is so "heavyweight", it's  
>>> necessary to do
>>> a lot of things at interrupt level just to get decent  
>>> performance.  In a
>>> well-designed microkernel system, context switching is fast  
>>> enough to do
>>> practically everything in userland with good performance.  This  
>>> improves
>>> robustness, debuggability, and CPU allocation.
>> Please provide examples that support this assertion.
>>>       FreeBSD:        Servers
>>>       NetBSD:         Portability
>>>       OpenBSD:        Security and Reliability
>>>       Dragonfly       Desktops
>>> Note that *none* of these actually set out to be optimized as a  
>>> router OS.
>>> However, OpenBSD's emphasis on security and reliability made it an
>>> attractive choice for firewall/router applications, which led to  
>>> greater
>>> developer attention to router issues, which led to increased use  
>>> as a
>>> router, etc.  etc.  Thus, it's become the de facto BSD of choice  
>>> for router
>>> applications, which receive a lot of developer attention,  
>>> including the
>>> creation of things like pf and carp.  In general, most *nix-based  
>>> routers
>>> use either Linux or OpenBSD, with the former choice being more  
>>> due to the
>>> "bandwagon factor" than real benefit.
>> Juniper runs FreeBSD.  True this is the control plane, not the
>> forwarding plane, but where are OpenBSD's "security" bennefits?
>> XORP primarily targets FreeBSD (4.11)
>> OpenBSD's team is far too willing to violate RFCs in their persuit to
>> "rule the world" in my (not so) humble opinion.  Have you looked  
>> at the
>> 'ntp' implementation they ship?
>> Have you *looked* at the mess they made of 802.11 support in OpenBSD?
>>> The effect of FreeBSD's "focus" is quite apparent in the 5.x fiasco.
>>> FreeBSD's main goal is to provide the best-performing server OS  
>>> on x86
>>> hardware, and it's done an excellent job of that (though recent  
>>> Linux
>>> versions are giving it a run for its money).  But in some cases,
>>> server-oriented improvements inflict collateral damage on the  
>>> routing code,
>>> as in 5.x.
>> Above you say they're not connected, now you say they are.
>>> There's no reason to suppose that this was a one-time thing -
>>> the FreeBSD developers are always going to pay more attention to  
>>> server
>>> issues and router issues, because *not* doing so would distract them
>>> from server improvements.
>> Nor can we say that OpenBSD will continue to be able to claim  
>> itself as
>> the "most secure".  Past performance is no indicator of future  
>> trends,
>> and all that.
>>> NetBSD would provide the largest choice of hardware platforms,  
>>> but it's not
>>> clear that there are any platforms significantly interesting as  
>>> routers
>>> that are supported by NetBSD and not OpenBSD.  Sure, there may be  
>>> a few
>>> people wanting to run m0n0wall on their dual-NIC toasters, but  
>>> that doesn't
>>> seem like a very good justification.  NetBSD is claimed to offer
>>> performance somewhere between FreeBSD and OpenBSD, but those are
>>> host-related claims.  And at least in V1.5.3, I found NetBSD to  
>>> have the
>>> worst installer and the most "pickiness" about the hardware  
>>> configuration
>>> of the three (never bothered trying Dragonfly).
>> But the 'installer' is a *host issue*, no?
>>> Thus, I consider OpenBSD to be the best choice, not primarily due to
>>> "better security" (although that's a factor), but because it's  
>>> the one
>>> branch of BSD where the developers take routing applications  
>>> seriously.
>>> The two main arguments that have been advanced against OpenBSD are:
>>> 1) Performance.  Some have made, and others have parroted, claims  
>>> that
>>> OpenBSD has substantially worse "networking performance" than  
>>> FreeBSD.  But
>>> AFAIK these are all based on *server* performance.  The only  
>>> *routing*
>>> performance comparison I've seen mentioned here was what Manuel  
>>> did before
>>> starting m0n0wall, which was of course several versions ago for both
>>> systems.  I've not seen any information on routing performance  
>>> comparisons
>>> between *current* versions of OpenBSD and FreeBSD.  Plus, due to  
>>> the "focus
>>> factor", one would generally expect that comparison to shift more  
>>> toward
>>> OpenBSD as the two systems evolve.
>> Just as OpenBSD's current advantages will tend to converge into  
>> FreeBSD.
>>> 2) Atheros support.  While the concept of a vendor-supplied HAL  
>>> makes
>>> sense, Atheros really screwed it up by providing it in binary- 
>>> only form
>>> (for a pretty lame reason AIUI).  If they had done it right - i.e.
>>> providing the HAL as CPU- and OS-independent C source, then there  
>>> would
>>> have been no issue.
>> I'm certain (as in, I've asked and gotten answers) that they would  
>> have
>> prefered this path as well.  It wasn't an option, due to the very  
>> simple
>> fact that it is *trivial* to tune the Atheros chipset *way* out of  
>> the
>> ISM (and U-NII) bands.   This, in effect means that an "open HAL"  
>> would
>> (not could, *would*) result in substantial interference issues with
>> users of licensed spectrum.   While I could detail the various  
>> license
>> holders who could scream (and sue) over this, I will point out one  
>> very
>> vocal set of "licensed" users who would pull every trick in the  
>> book the
>> second an "open HAL" resulted in detectible interference:   HAM Radio
>> Operators (especially the ARRL).  You see, the (nearly world-wide)  
>> 13cm
>> band runs from 2304MHz to 2450Mhz.
>>> As it is, some people grudgingly accepted the
>>> "official" HAL, while someone else developed an open-source  
>>> replacement HAL
>>> by reverse engineering Atheros's, and that was introduced in the  
>>> latest
>>> version of OpenBSD.
>> And it... doesn't work.  Its total crap.  Have you tried it? Moreover
>> its clear (for those of us who can look at both the openbsd and  
>> "closed"
>> HAL source code that there is clear copyright infringement in the
>> (supposed) "reverse engineered" HAL in OpenBSD.
>> Want to get (your customer's) sued?  Run the "reverse engineered"  
>> HAL.
>>> You'll find things of the form "It's evil because it's
>>> based on reverse engineering" and "Is this really legal", as well  
>>> as "When
>>> is OpenBSD's open-source HAL going to be ported to our platform  
>>> so we can
>>> junk the binary-only crock?".  I have yet to see any actual  
>>> behavioral
>>> comparisons between the two.
>> I have.  I've run them.  We *actively refuse* to support people  
>> running
>> the "open HAL" code on products we sell.
>>> Of course the best scenario would be if the open-source HAL  
>>> became sufficiently popular to shame Atheros into providing the  
>>> "official" HAL in source form.  :-)
>> Its not going to happen.   First it would have to work, then the  
>> authors
>> of the "open source HAL" would end up in court, having to defend
>> themselves from a  copyright infringement suit.  Then they would  
>> end up
>> in court having to defend themselves from the wrath of "spectrum
>> licensees" and the regulatory agencies that granted (or sold) those
>> licensees.
>>> Note that if there really were a significant advantage in using  
>>> the binary
>>> HAL in OpenBSD, it could be used with a "wrapper" in the same  
>>> manner as it
>>> is in FreeBSD.
>> Too bad OpenBSD screwed with net80211, or one could merely  
>> "replace" one
>> with the other.
>>> One could even do a side-by-side comparison of two builds
>>> of OpenBSD, one with the binary HAL and one with the open-source  
>>> HAL.  But
>>> I wouldn't go to that much trouble without determining that the  
>>> effort is
>>> warranted.
>> I've run OpenBSD and FreeBSD on two CF cards on the same hardware
>> (soekris and wrap boards) with the same radio cards.   OpenBSD has a
>> hard time even maintaing associations.
>>> More generally, it's important to make a distinction between what  
>>> devices
>>> are "officially" supported by OpenBSD, and what devices are  
>>> supported by
>>> third-party drivers.  OpenBSD is more rigorous about the  
>>> definition of
>>> "open-source" than some other systems, and hence can't officially  
>>> include
>>> drivers that aren't fully open-source.  I'd expect the same to be  
>>> true of
>>> Linux due to GPL requirements.
>> Non-GPL drivers will get more difficult to incorporate into linux  
>> over
>> time.   this is especially true as the strings are
>> tightened on object-only drivers that have to semi-obviously make
>> reference (and incorporate) GPL liccensed header files, data  
>> strcutures,
>> etc.
>> A carefull analysis of the Atheros-supplied HAL will show an elegant
>> method for avoiding that mess.
>> But I digress.
>>> While I don't consider an open-source
>>> system to be "tainted" by the inclusion of binary *firmware* to  
>>> be loaded
>>> into peripherals, if any code executed by the CPU isn't open- 
>>> source, then
>>> the system can't be honestly described as open-source without a  
>>> suitable
>>> disclaimer.  And one has to question the wisdom of including  
>>> "black-box"
>>> code in a firewall.
>>> It's also worth noting that binary driver code is at the very least
>>> CPU-specific, making it a major hassle for an OS that supports  
>>> multiple CPU
>>> architectures.  FreeBSD tolerates this more easily due to being,  
>>> for all
>>> practical purposes, x86-only, but other OSes have more difficulty  
>>> with it.
>>> And there seems to be quite a bit of interest in running m0n0wall on
>>> non-x86 router hardware.
>> The Atheros HAL object is identical (even in binary form) between  
>> linux
>> on x86, netbsd on x86 and freebsd on x86.  There is an OS-dependent
>> 'wrapper' for it.
>> For netbsd, mach (as used in Apple's product) and linux the same  
>> is true
>> for PowerPC (albeit with big/little endian being different
>> architectures. 64/32 bit would be an issue if NetBSD and linux didn't
>> both run the kernel in 32-bit mode.  (Mach too.))
>> Reducing to NetBSD and Linux, this is still true for
>> ARM9 (big and little endian)
>> Xscale
>> MIPS big and little endian
>> Your point (as far as it relates to the "closed source" (Atheros
>> supplied) HAL is moot.
>>> OpenBSD and NetBSD are sufficiently similar internally that  
>>> drivers (and
>>> even in some cases kernel patches) can usually be ported from one  
>>> to the
>>> other fairly easily.  Thus, any device supported by NetBSD can be  
>>> at least
>>> unofficially supported by OpenBSD without too much difficulty.   
>>> FreeBSD
>>> is sort of the "odd man out" in this department.
>> This is mostly because Theo "forked" OpenBSD after the NetBSD guys
>> kicked him out for being ... an asshole.
>> Only *then* did we start to get "security" as a focus.
>>> The bottom line is that the two major arguments against OpenBSD  
>>> are at the
>>> very least lacking in data, if not simply wrong.
>> Try again.  Please.
>>> Issues with OpenBSD:
>>> --------------------
>>> Here are some issues I'm aware of that would need to be addressed  
>>> in moving
>>> to OpenBSD:
>>> 1) FTP NAT performance:  Unlike IPFilter's in-kernel FTP proxy,  
>>> pf relies
>>> on the userland ftp-proxy program.  As far as the control  
>>> connection goes,
>>> this is a big win, since handling the control connection at the  
>>> packet
>>> level is a horribly kludgy and fragile mess.  So much so that  
>>> Cisco has
>>> fixed bugs in that area within the past year.  And control- 
>>> connection
>>> performance is a non-issue.  However, ftp-proxy also proxies the  
>>> data
>>> connection through userland, which involves a significant  
>>> performance
>>> penalty.  It can avoid that in two out of the four combinations of
>>> client/server and active/passive, but not the remaining two.   
>>> There's an
>>> (unofficial) alternative called "ftpsesame", which manipulates  
>>> the filter
>>> tables to forward the data connection properly, but it has a  
>>> horrible
>>> packet-oriented bpf-based kludge for the control connection.   
>>> Ideally, one
>>> would like a hybrid of the two approaches, but AFAIK none such  
>>> exists ATM.
>>> Note that this is not strictly an OpenBSD/FreeBSD issue, but  
>>> rather a
>>> pf/IPFilter issue, so pfsense is presumably already living with  
>>> the extra
>>> overhead, and since practically everyone seems to agree that it  
>>> makes sense
>>> to move to pf instead of IPFilter, it's an issue regardless of OS  
>>> choice.
>>> Fixing it might be a bit easier in OpenBSD, simply because it's  
>>> the "native
>>> habitat" of both pf and ftp-proxy, without "compatibility layers"  
>>> getting
>>> in the way.
>> This is just arrogant.  pf is fully supported in FreeBSD 6 (and  
>> above?).
>>> Advantages of OpenBSD:
>>> ----------------------
>>> 1) Strong orientation toward router-related issues.
>>> 2) Proactive security.  Some have described this as a "solved  
>>> problem",
>>> but it doesn't remain "solved" as long as the code keeps changing.
>> But as this code chances, the other two "BSDs" tend to pick up the
>> changes, "quickly".
>> Not all security issues are addressed first in OpenBSD.
>>> 3) Most aggressive development in cryptographic areas, including  
>>> support
>>> for hardware accelerators.
>> Nevermind that Sam Leffler is the primary author of many of these,  
>> and
>> he develops them for FreeBSD first.
>>> 4) Widely desired features like pf and CARP in their "native  
>>> habitat".
>> Now a non-issue.
>>> 5) Ability to run on non-x86 hardware.
>>> 6) IPv6 support that's actually usable.  At least as recently as  
>>> 5.3,
>>> FreeBSD had conflicts between IPv6 and FAST_IPSEC which could  
>>> cause kernel
>>> panics.  Although IPv6 isn't exactly taking the world by storm, it
>>> definitely needs to be in any reasonable plan for the future.
>> Could we discuss 6.0 or better, please.  Especially given that  
>> neither
>> m0n0wall or pfSense are likely to be based on any 5.x release of  
>> FreeBSD?
>>> 7) Improved IPsec support, including NAT-T.
>>> 8) Development cycle based on conservative goals, where new  
>>> releases have
>>> a high probability of being adoptable without significant problems.
>> Anyone who has done software development for as long as either of us
>> knows that this kind of statement can't stand for long.
>> Jim
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: m0n0wall dash dev dash unsubscribe at lists dot m0n0 dot ch
>> For additional commands, e-mail: m0n0wall dash dev dash help at lists dot m0n0 dot ch