On Tue, 18 Jan 2005 12:31:09 -0700, Eric Collins <eric at tawifi dot com> wrote:
> Greetings List,
> 
> I just wanted to get some recommendations on what other people are using
> as far as tweaks to m0n0 for high throughput systems.
> What we have is a customer with a full DS3 (45mbit) plus a backup DS3
> (BGP) that we are re-building a webfarm for and we would like to place
> m0n0 as a filtering bridge in front of the farm. Here is a URL of the
> rough network layout as it currently sits http://www.tawifi.com/ds3net.jpg
> 
> We have m0n0 in the diagram where it will be placed, currently it is a
> direct connection and the average throughput as of 10 minutes ago was
> 24Mbps sustained, and it runs at this rate pretty consistantly 24/7 but
> it does burst to 30-35 quite regularly in peak usage.
> 
> The server that will be used for m0n0 has these specs:
> AMD AthonXP 1700
> 512MB ECC Ram
> 2 Intel 10/100/1000 NICs (dont know the model right now)
> 1 3Com 10/100 (dont know the model uses xl driver)
> 512MB SanDisk CF
> 

You might need to up the kern.ipc.nmbclusters sysctl, which would have
likely fixed the problem you had with OpenBSD as well, though I'm not
familiar enough with Open to tell you definitively.

From 'man tuning':

--
     kern.ipc.nmbclusters may be adjusted to increase the number of network
     mbufs the system is willing to allocate.  Each cluster represents approx-
     imately 2K of memory, so a value of 1024 represents 2M of kernel memory
     reserved for network buffers.  You can do a simple calculation to figure
     out how many you need.  If you have a web server which maxes out at 1000
     simultaneous connections, and each connection eats a 16K receive and 16K
     send buffer, you need approximately 32MB worth of network buffers to deal
     with it.  A good rule of thumb is to multiply by 2, so 32MBx2 = 64MB/2K =
     32768.  So for this case you would want to set kern.ipc.nmbclusters to
     32768.  We recommend values between 1024 and 4096 for machines with mod-
     erates amount of memory, and between 4096 and 32768 for machines with
     greater amounts of memory.  Under no circumstances should you specify an
     arbitrarily high value for this parameter, it could lead to a boot-time
     crash.  The -m option to netstat(1) may be used to observe network clus-
     ter use.  Older versions of FreeBSD do not have this tunable and require
     that the kernel config(8) option NMBCLUSTERS be set instead.
--

add a line like the following to the /boot/loader.rc on the image
set kern.ipc.nmbclusters=32768

That'd take 64 MB RAM.  With 512 MB RAM and m0n0wall, you could easily
set it way higher than that, but setting it arbitrarily high may cause
problems as stated above.

The default on FreeBSD and m0n0wall is 1024, which is fine until you
start pushing a ton of connections.

-Chris