[ previous ] [ next ] [ threads ]
 
 From:  =?WINDOWS-1252?B?lSCV?= <googl3meister at gmail dot com>
 To:  Gavin Pidgley <gavin at gadago dot net>
 Cc:  Rolf Sommerhalder <rolf dot sommerhalder at alumni dot ethz dot ch>, m0n0wall at lists dot m0n0 dot ch
 Subject:  Re: [m0n0wall] 1.2b8 hanged: all mbufs exhausted
 Date:  Tue, 28 Jun 2005 19:30:15 +1000
On 6/28/05, Gavin Pidgley <gavin at gadago dot net> wrote:

> > On 6/28/05, Rolf Sommerhalder <rolf dot sommerhalder at alumni dot ethz dot ch> wrote:
> > > My generic-PC 1.2b8 runs as a filtering bridge on a 4 Mbit/s symmetric
> > > connection and after 3 days uptime shows in the syslog:
> > >     /kernel: All mbufs exhausted, please see tuning(7).
> > >
> > > The bridge apparently still works, even though log outputs is severly
> > > limited to a few lines only (I had set it at 200 lines).
> > >
> > > output of 'netstat -m' is:
> > >
> > > $ netstat -m
> > > 9468/9536/9536 mbufs in use (current/peak/max):
> > >         100 mbufs allocated to data
> > >         9368 mbufs allocated to packet headers
> > > 69/198/2384 mbuf clusters in use (current/peak/max)
> > > 2780 Kbytes allocated to network (38% of mb_map in use)
> > > 46 requests for memory denied
> > > 0 requests for memory delayed
> > > 6 calls to protocol drain routines
> > >
> > > Any hints of ehat I should try? (If needed, I can compile the kernel and
> > > produce a new custom image, no problem.)
> > >
> > > Rolf
> > >
> >
> > You need to up the max number of network memory buffers, using:
> > > sysctl -w kern.ipc.nmbclusters=X
> >
> > For a value for X, considering memory and usage of the machine, here
> > is the relevant bit from tuning(7):
> >      kern.ipc.nmbclusters may be adjusted to increase the number of network
> >      mbufs the system is willing to allocate.  Each cluster represents approx-
> >      imately 2K of memory, so a value of 1024 represents 2M of kernel memory
> >      reserved for network buffers.  You can do a simple calculation to figure
> >      out how many you need.  If you have a web server which maxes out at 1000
> >      simultaneous connections, and each connection eats a 16K receive and 16K
> >      send buffer, you need approximately 32MB worth of network buffers to deal
> >      with it.  A good rule of thumb is to multiply by 2, so 32MBx2 = 64MB/2K =
> >      32768.  So for this case you would want to set kern.ipc.nmbclusters to
> >      32768.  We recommend values between 1024 and 4096 for machines with mod-
> >      erates amount of memory, and between 4096 and 32768 for machines with
> >      greater amounts of memory.  Under no circumstances should you specify an
> >      arbitrarily high value for this parameter, it could lead to a boot-time
> >      crash.  The -m option to netstat(1) may be used to observe network clus-
> >      ter use.  Older versions of FreeBSD do not have this tunable and require
> >      that the kernel config(8) option NMBCLUSTERS be set instead.
> >
> > ie: it depends what you're using it for and how much memory it has.
> 
> Does anyone have a recommended value for network buffers on a wrap board
> with 64MB of memory?
> 

Not from personal experience, sorry - there is an FAQ entry also: 
http://www.m0n0.ch/wall/docbook/faq-high-performance-tweaks.html

where it describes that m0n0 is set to 1024 by default although that
seems to contradict what I see in your output - so check how much free
RAM you have and start increasing - 32768 will take the whole 64MB
leaving no room for OS (assuming 16k rx and tx buffers in the
application), so if you have 25% free RAM under normal conditions, =>
32768 / 4 => 8192 is your new number.  Personally I'd start with a
much lower value (1536 [50% of the default] up to 2048 [double the
default]) and keep a very close eye on it...

Can anyone else comment on the default value (1024) authoritatively,
as I have no time to look at source right now?

--g'luck
gm