** I'm not speaking on behalf of my current employer **
Manuel and friends,
First, thank you for m0n0wall. It really rocks, no two ways about it.
I've lurked on this list for about a year, but I think it might be
time to mention something my colleagues and I worked on a few months
back as a research project.
In a nutshell, we are already running m0n0wall on top of Linux.
I'm not asking y'all to believe me sight unseen, since it sounds
pretty far-fetched. For all you know, I just might be a very good
liar. m0n0 is after all, pretty wedded to the system commands and
configuration files you'd find on a BSD box. But using a few cunning
stunts, I am claiming I found a way to do it.
Our prototype doesn't offer the full m0n0wall functionality, but as a
proof of concept the basics are certainly working. We have a
workable, useful Linux-based router running m0n0wall as the GUI. And
with the exception of stuff Linux just doesn't support, our method can
with little effort be extended to do under Linux anything that
m0n0wall can do on FreeBSD.
Surprisingly, the diff against the m0n0 code is less than 100 lines.
(The magic to allow all this isn't part of m0n0wall, and hence, isn't
part of the 100 lines)
Some other points of interest:
- Since the target platform is commodity router hardware, we are
running it on low footprint boxes: 8M for code/config, and 16M of RAM.
So far this is fine. We can get 30-35 megabytes/sec through the
ethernet, no worries, while using the GUI. (Hope the memory
requirements don't grow too much! :-) )
- We run it on cut down PCs (i386) and on ARM boards.
- For PCs we use a bootable CD for the code, and a floppy for the
config. We use the kernel boot-time option to limit available memory
to 16M, and use NFS mounting of the web server pages to speed up
- For ARM boards we use onboard direct mapped flash. The reason we
have to run it on top of Linux rather than FreeBSD is that there are
proprietary Linux-only drivers for things on the ARM board like the
network and USB interfaces. Also, all our experience is with embedded
- Our operating environment is based around a snapshot of the uClibc
"buildroot" system from about this time last year. The kernel is
2.4.20 for the ARM boards, and 2.6.11 for the i386.
- For a webserver, I hacked the busybox httpd to be able to call PHP
properly. So we don't have (or need) a standalone webserver program.
We are calling PHP as a CGI binary. We have considered using DSOs or
shared libs but haven't seen a need for it so far.
- We are running m0n0 on PHP5, compiled with most of the standard
stuff removed. All that's really needed is the regexp and XML stuff.
The exe is about 800k, and the RSS of a running PHP request is about
- PHP is running with all warnings turned on. That's really noisy as
m0n0wall thinks it's ok to evaluate vars that have never been set. I
got someway towards rewriting m0n0 to take special care to either set
a var first or check if it's defined before evaluation, but gave up.
The divergence is just too expensive. In the end, I hacked PHP to
still run with all warnings, but commented out the sections in the PHP
C code that check for those particularly noisy but harmless warnings
that m0n0wall on PHP5 generates.
When I looked for no fee/no royalty GUIs, m0n0wall was simply the best
choice available. And working out the magic for Linux was a heap of
Ok, having said that, how does this relate to m0n0wall's future?
I'd like to share with you what our thoughts are as far as where we
would like to take m0n0wall. I'm not suggesting the m0n0wall project
takes any of these onboard, since they are ideas to solve *our*
problem, not the general m0n0 problem. I would also note that as far
as I'm able, I disclaim all rights to the IP of ideas I mention here.
- While we really don't expect Linux to be chosen as the next m0n0
host OS, we will continue to run m0n0 on top of Linux. If and when we
resume our m0n0 efforts (tied up with other jobs at the moment), we'll
be adding support for the parts of m0n0 that we haven't got working
under Linux yet. The cost of getting it to work under Linux is less
than the cost of improving the other on-Linux-but-feature-incomplete
- Perhaps our biggest problem is that we have a need for multiple
UIs: Web, UPnP, CLI, telnet (menu-based) and SNMP. How to i) support
these and ii) handle the data coherency problems is something we've
been thinking hard about.
- Separation of the UI front end from the core logic back end is a
good idea. Not just to support multiple UIs or to aid in program
structure, but to solve the coherency problem: While it's desirable
to have multiple ways of interfacing, the interfaces should only
operate on one set of data, ie, many-to-one. It's not a good idea for
all the UIs to be madly writing out their own versions of
configuration files! The back end should provide interfaces to
retrieve and change system settings, and handle the responsibility of
calling the usual helpers (such as ifconfig), writing config files for
daemons, etc, as well as *transparently* persisting those changes into
config.xml. It should also arbitrate if multiple interfaces are
active at the same time.
- While our existing efforts have been intentionally minimally
invasive (we didn't want to diverge too much from the existing code),
supporting a frontend/backend architecture, and indeed multiple UIs,
would require a rather major restructuring of the code. If and when
we continue our efforts with m0n0wall, and assuming m0n0wall hasn't
already progressed in this direction, we'll probably do this rewrite
ourselves. Since such a large divergence would mean pain for us, we
would consider releasing our changes for this part.
I want to acknowledge the good ideas others have had in the past few
days: Manuel, Chris Buechler, Michael Hanselmann, Marc Fargas and
Peter Curran. You guys have all independently mentioned parts of our
plan. That's encouraging as far as validation goes.
Some more ideas, which :
- Regarding languages, we are happy with PHP. Disk and RAM-wise,
once you've taken the memory hit of the PHP interpreter, running the
actual PHP page is very very low cost in terms of memory. I believe
the expressiveness, clarity and density of PHP is very good, and
things such as the string handling make it much easier to write secure
maintainable programs. Michael Hanselmann suggested that PHP be
retained because it's what makes m0n0 unique. That's preserving PHP
simply for uniqueness' sake. I'd suggest a more practical way of
looking at it: m0n0 is uniquely customizable *because* it's written
in PHP. And an architectural update changes nothing about that
And perhaps the biggest reason to not change languages:
"The idea that new code is better than old is patently absurd. Old
code has been used. It has been tested. Lots of bugs have been found,
and they've been fixed"
It's really worth reading that article. New code may look clearer,
but old code has warts because bugs were fixed. New code won't get to
the same level of functional polish for a long time. I would suggest
salvaging as much as possible from the old code, even as far as
changing the architecture gradually over time.
If we want to take advantage of OOP, there's PHP5. While PHP4's OO
was a joke, PHP5's OO is a definite asset.
- As a communication mechanism between front ends and the back end,
we are considering using XMLRPC. Very light weight (lighter than
SOAP), easy to debug and well supported, especially in PHP.
Multi-language, cross platform, no problems. While any RPC-using
program tends to be complicated by the possibility of unreliable comms
(maybe the server goes away!), things in the m0n0wall case are
simplified somewhat by having the server and the client on the same
machine. The server can just be assumed to be there all the time. For
the remainder of problems, PHP5's exceptions help keep code clear,
while still handling problems gracefully.
I see more or less two strategies for defining the interface.
- Treat the interface as merely setters and getters on *data*.
It's up to the back end to translate this into daemon config file
rewrites, daemon restarts, etc. This is similar to the data model
that SNMP presents, although it would be done via XMLRPC. UI code
might consist of setters and getters like:
backend_set("interfaces.fxp0.mtu", 1500) or: $v =
- The interface consists of objects that represent the system.
XMLRPC calls on methods of those objects do the setting and getting.
The keys to use (in the first option) or the classes (in the second)
could closely model the schema of the XML file. The backend could
consist of actions which get linked to certain nodes in the XML config
file tree. These actions get triggered when the setters and getters
are called on a particular node.
The second of these is the most OO pure, since you're calling methods
on interface objects, but the first of these is probably the easiest.
Manuel, you mentioned "without relying on an external PHP CGI binary".
Especially in embedded systems, you have to be able to handle the
worst-case everything-running-at-once situation without crashing. But
for performance, it's also desirable to dump things if you can.
My idea is to have the (minimal) webserver do two jobs. The first is
to listen to requests for UI pages from a browser to the web-based
front end. The second is to listen to XMLRPC requests from front
ends, then spawn the PHP backend. That way, we can get away with only
having the back-end in memory when it's needed, while still
permanently listening to XMLRPC and GUI requests (via the webserver).
We may also add throttling to prevent simultaneous PHP backends, or do
some sort of back-end "hang-around" mechanism. (Marc Fargas' email
said much the same thing).
> I don't see how you could write a multi-threaded network-enabled daemon with PHP... maybe somebody
can show us.
Here's one solution. While it's ok to have multiple front ends, have
a mechanism to enforce only one backend instance. That helps with
data coherence. Then have a design rule for the backend that it
doesn't do any operation that will block for more than some period of
time, maybe one second. If an operation might take longer than one
second (for example, invoking pppd), it should be forked, and there
should be a separate interface method for retrieving the current
status of that operation. Web pages can be written to do
auto-refreshes, CLIs can block, or whatever, but the back end does not
block for too long.
(Hmm, I also see that PHP has the "socket_select()" wrapper for the
select() system call. Netscape used this system for years to "fake"
A final comment from Chris Buechler: "I bet at least 25% of the user
base thinks m0n0wall runs on Linux"
Ours does! :-)
Anyway, you're all probably either bored or outraged right about now,
so I'll get off my soapbox. Thank you, I look forward to comments.