[ previous ] [ next ] [ threads ]
 
 From:  mtnbkr <waa dash m0n0wall at revpol dot com>
 To:  m0n0wall <m0n0wall at lists dot m0n0 dot ch>
 Subject:  Re: [m0n0wall] 1.3.b11 lockups
 Date:  Fri, 16 May 2008 14:01:15 -0400
Thanks for any effort you can offer on this, Manuel.



Manuel Kasper wrote:
> On 16.05.2008, at 19:05, mtnbkr wrote:
> 
>> I have seen not lockups, but random kernel panics and reboots on both 
>> the WRAP and a new ALIX 3 port... Same site, same config same issues.
>>
>> In my case it appeared that the state table might have been getting 
>> overrun - if memory serves me.
>>
>> I had a serial cable connected to the serial port, and at the same 
>> time the m0n0wall stopped logging to my remote syslog server I had 
>> seen errors on the serial console - followed by kernel panic and 
>> followed shortly by reboot.
>>
>> Perhaps your issue, like mine is less ALIX-specific and more "FreeBSD 
>> plus your site specific traffic" related.
>>
>> Manuel?   Your thoughts - on my past post regarding these errors and 
>> reboots?
> 
> Reading your post from Mar 19, I get the impression that your issue 
> doesn't have much if anything to do with the lockups that Luke is 
> reporting. Instead, for some reason, your system keeps running out of 
> memory for the kernel. Perhaps there's a memory leak somewhere (in 
> ipfilter?), but I also wonder why your log messages seem to indicate 
> that you have over 20000 concurrent connections at times. Could be heavy 
> P2P usage or something similar, though.


The interesting thing to me is that typically, almost all outbound 
traffic must go out to the Internet via an internal squid web cache 
server (with filtering by Dans Guardian on same box)

Just about everything else is blocked and dropped (not rejected) by the 
m0n0wall. There is no P2P traffic allowed. It's a pretty strict setting 
in that sense.

There are a few rules to allow smtp/http/https/pop3 from the Internet to 
the email server(s) on DMZ network (OPT1 on WRAP) as well as a few rules 
to allow specific types of traffic from inside to the DMZ (backup 
server, outbound SMTP from internal servers to email server on DMZ etc)

> Some "vmstat -m" output, generated after the firewall has seen some 
> heavy use for a while, could give us a pointer in the right direction... 
> Also, "ipfstat -s" output would be interesting. Try this:

OK. This particular box has been up for a little over 9 days right now..
vmstat and ipfstat outputs follow:


$ /tmp/vmstat -m
          Type InUse MemUse HighUse Requests  Size(s)
     proc-args    14     2K       -      332  32,64,128,256
        zombie     0     0K       -     3215  128
   ata_generic     1     1K       -        1  1024
       ithread    53     5K       -       53  16,64,128
     ad_driver     1     1K       -        1  32
        linker    47    62K       -       66  16,32,256,1024,4096
         lockf     2     1K       -        2  64
       ata_dma     2     1K       -        2  128
          temp 12026  7684K       - 47613865 
16,32,64,128,256,512,1024,2048,4096
        devbuf   131   182K       -      132 
16,32,64,128,256,1024,2048,4096
        module   186    12K       -      186  64
      mtx_pool     1     8K       -        1
       subproc   105   207K       -     3320  256,4096
          proc     2     1K       -        2  512
       session     7     1K       -        7  128
          pgrp     7     1K       -        7  64
          cred     8     1K       -     8205  128
       uidinfo     3     1K       -        3  32,128
        plimit     3     1K       -       23  256
     sysctltmp     0     0K       -        8  16,32
     sysctloid   888    27K       -      888  16,32,64
        sysctl     0     0K       -       59  16,32
          umtx    70     5K       -       70  64
       entropy  1024    64K       -     1024  64
        bus-sc    21     3K       -      220  16,64,128,256,512,2048,4096
           bus   549    57K       -     1114  16,32,64,128,1024
       devstat     4     9K       -        4  16,4096
  eventhandler    43     3K       -       43  32,128
          kobj    91   182K       -      110  2048
          rman    53     4K       -      423  64
          sbuf     0     0K       -       90  16,32,512,1024
        DEVFS1    91    23K       -       93  256
        DEVFS3   103    13K       -      106  128
     taskqueue     7     1K       -        7  16,128
        Unitno     9     1K       -      149  16,64
           iov     0     0K       -       90  16,64
      ioctlops     0     0K       -  1484031  16,32,64,128,512,1024
           msg     4    25K       -        4  1024,4096
           sem     4     7K       -        4  512,1024,4096
           shm     1    12K       -        1
          ttys   137    18K       -      309  128,1024
      mbuf_tag     0     0K       -        5  32
           pcb    45     6K       -       51  16,32,64,2048
        soname     2     1K       - 12005467  16,32,128
    BIO buffer    31    31K       -       86  1024
      vfscache     1    32K       -        1
cluster_save buffer     0     0K       -       12  32,64,128
      VFS hash     1    16K       -        1
        vnodes     1     1K       -        1  128
   vnodemarker     0     0K       -    50667  512
         mount    30     1K       -      125  16,32,64,128,2048
           BPF    26    10K       -       26  64,256,4096
   ether_multi    17     1K       -       20  16,32,64
        ifaddr    50    17K       -       50  32,256,512,2048
         ifnet    23    23K       -       24  256,512,1024
         clone     6    24K       -        6  4096
        arpcom     3     1K       -        3  16
            lo     1     1K       -        1  16
         DEVFS    27     1K       -       28  16,128
CAM dev queue     1     1K       -        1  64
          GEOM    34     4K       -      184  16,32,64,128,256,512,1024
        isadev    22     2K       -       22  64
      routetbl   158    35K       -      322  16,32,64,128,256
     CAM queue     3     1K       -        3  16
  netgraph_msg     0     0K       -      245  64,128,256
netgraph_node    52    13K       -       52  256
netgraph_hook    32     4K       -       66  128
      netgraph     3     1K       -        3  32
netgraph_iface    17     2K       -       17  64
  netgraph_ppp    16    32K       -       16  2048
netgraph_sock    16     1K       -       16  64
netgraph_path     0     0K       -      245  16
      in_multi     4     1K       -        4  32
   Export Host     1     1K       -        1  1024
     hostcache     1    24K       -        1
       CAM SIM     1     1K       -        1  64
      syncache     1     8K       -        1
   inpcbpolicy     8     1K       -     9475  16
   ipsecpolicy    16     4K       -    24578  256
        crypto     3     1K       -        3  128,512
         xform     0     0K       -      532  16,32
      p1003.1b     1     1K       -        1  16
        newblk     1     1K       -        1  256
      inodedep     1    16K       -        1
       pagedep     1     4K       -        1  4096
     UFS mount     6     9K       -       15  256,2048
       UMAHash     1     1K       -        3  256,512,1024
    CAM periph     1     1K       -        1  128
       CAM XPT     5     1K       -        7  16,64,512
           LED     3     1K       -        3  64
          cdev    17     3K       -       17  128
     VM pgdata     1     8K       -        1
       MD disk     1     2K       -        1  2048
         sigio     1     1K       -        1  32
     file desc    50    13K       -     3265  16,256,512
     legacydrv     3     1K       -        3  16
      nexusdev     2     1K       -        2  16
          kenv    11     5K       -       11  16,32,4096
        kqueue     0     0K       -        4  256,1024
   IpFw/IpAcct     3     1K       -       33  64,128



$ ipfstat -s
IP states added:
	2508508 TCP
	4845593 UDP
	50857 ICMP
	658720977 hits
	13528956 misses
	0 bucket full
	0 maximum rule references
	0 maximum
	0 no memory
	3417 bkts in use
	3648 active
	4894017 expired
	2507293 closed
State logging enabled

State table bucket statistics:
	3417 in use	
	93% hash efficiency
	7.97% bucket usage
	0 minimal length
	4 maximal length
	1.068 average length

TCP Entries per state
      0     1     2     3     4     5     6     7     8     9    10    11
      0     3     2     0   369     1     7     0     0     0   714   119


--
Bill Arlofski
Reverse Polarity, LLC