[ previous ] [ next ] [ threads ]
 
 From:  "Kasper Pedersen" <m0n0list dash kkp2 at kasperkp dot dk>
 To:  <m0n0wall at lists dot m0n0 dot ch>
 Subject:  Post mortem autopsy on a dead monowall
 Date:  Tue, 14 Nov 2006 19:48:13 +0100
Synopsis:
  One of my monowalls crashed twice. I replaced it with a virtualized 
monowall, and it crashed too.
  I then dumped the entire physical memory to disk and had a look at it.
  Strange things were found lurking in the dump.


And the longer version:

With two crashes on the same otherwise-reliable monowall in 24 hours, I 
switched it to a virtualized one, wondering if my hardware (LEX embedded 
board) had gone bad. Half a day later the VM'ed mono became unresponsive and 
consumed 100% CPU. I froze the VM (and I still have the frozen one), copied 
the memory image to a Linux box, and used strings and grep to find the log: 
On the first two crashes on real hardware the syslog showed nothing, so I 
worked from the idea that it might be outputting an error message, but 
didn't live long enough to get it sent off to the syslog server.

The result of 'strings freebsd.vmem|grep "Nov 14">f2.txt' is at 
http://n1.taur.dk/deadmono/f2.txt
Here's the interesting part, the first 5 entries are purely for continuity:
----------------excerpt-----------------
Nov 14 02:47:25 virtualmono ipmon[80]: 02:47:24.990317 lnc0 @0:22 b 
192.168.10.17,59838 -> 192.168.10.255,3052 PR udp len 20 547 IN
Nov 14 02:47:49 virtualmono ipmon[80]: 02:47:48.308027 lnc0 @0:22 b 
192.168.10.245,137 -> 192.168.10.255,137 PR udp len 20 78 IN
Nov 14 02:47:49 virtualmono ipmon[80]: 02:47:48.317109 lnc0 @0:22 b 
192.168.10.6,137 -> 192.168.10.255,137 PR udp len 20 78 IN
Nov 14 02:47:49 virtualmono ipmon[80]: 02:47:48.667763 lnc0 @0:22 b 
192.168.10.242,137 -> 192.168.10.255,137 PR udp len 20 78 IN
Nov 14 02:47:51 virtualmono ipmon[80]: 02:47:50.568596 lnc0 @0:22 b 
192.168.10.17,59838 -> 192.168.10.255,3052 PR udp len 20 576 IN
Nov 14 02:47:54 virtualmono ipmon[80]: 02:47:53.950887 lnc1 @0:23 b 
211.205.0.60 -> 80.163.183.190 PR icmp len 20 56 icmp unreach/port for 
80.163.183.190,30573 - 211.205.0.60,1026 PR udp len 20 514 IN
Nov 14 02:47:54 virtualmono ipmon[80]: 02:47:53.951917 lnc1 @0:23 b 
211.205.0.58 -> 80.163.183.190 PR icmp len 20 56 icmp unreach/port for 
80.163.183.190,30571 - 211.205.0.58,1026 PR udp len 20 514 IN
Nov 14 02:47:54 virtualmono ipmon[80]: 02:47:53.952054 lnc1 @0:23 b 
211.205.0.61 -> 80.163.183.190 PR icmp len 20 176 icmp unreach/port for 
80.163.183.190,30574 - 211.205.0.61,1026 PR udp len 20 514 IN
Nov 14 02:47:54 virtualmono ipmon[80]: 02:47:53.979465 lnc1 @0:23 b 
211.205.0.55 -> 80.163.183.190 PR icmp len 20 56 icmp unreach/port for 
80.163.183.190,30568 - 211.205.0.55,1026 PR udp len 20 514 IN
--------------end excerpt----------------
According to VMWare's log, it died 02:48:02. The windows box is about 4 
seconds fast, and it takes VMWare a few seconds to notice it, so that timing 
is about right.
The new thing in this regard is the last 4 lines. Four address in the same 
range, ICMP destination unreachable, even though there was no outgoing 
traffic. At this point I thought it to be a reflection scan, but the box is 
not suitable for that. And one of the packets has a funny size.

Since I had failed to sniff the external interface I kludged up a program to 
search the dump for IP packets where the destination address matched the 
external interface, and this is what I found:
(the output is at http://n1.taur.dk/deadmono/packetsearch.txt and contains a 
1500 byte dump for each packet)
This is raw data, prefixed with source IP for readability
------------------
Some TCP
219.95.230.76  45 00 00 30 52 D9 40 00 70 06 ED E0 DB 5F E6 4C 50 A3 B7 BE
219.95.230.76  45 00 00 30 53 31 40 00 70 06 ED 88 DB 5F E6 4C 50 A3 B7 BE
219.95.230.76  45 00 00 30 53 7F 40 00 70 06 ED 3A DB 5F E6 4C 50 A3 B7 BE
201.215.230.149  45 00 00 30 98 83 00 00 75 06 F4 75 C9 D7 E6 95 50 A3 B7 BE
201.215.230.149  45 00 00 30 99 4A 00 00 75 06 F3 AE C9 D7 E6 95 50 A3 B7 BE
201.215.230.149  45 00 00 30 9A D8 00 00 75 06 F2 20 C9 D7 E6 95 50 A3 B7 BE
icmp echo's (ping)
83.190.157.119  45 00 00 3D 6F A8 00 00 76 01 DB 80 53 BE 9D 77 50 A3 B7 BE
83.190.157.119  45 00 00 3D 71 2B 00 00 76 01 D9 FD 53 BE 9D 77 50 A3 B7 BE
this one looks corrupted, protocol 0xB1 is unknown.
85.140.42.212  46 D5 04 00 44 00 00 00 80 B1 63 C1 55 8C 2A D4 50 A3 B7 BE

Now comes the fun ones (if you can read IP headers):

211.205.0.66  45 C0 1E 02 F1 F4 00 00 35 01 B4 B9 D3 CD 00 42 50 A3 B7 BE
03 01 92 27 00 00 00 00 45 00 02 02 F7 0F 00 00
This is an icmp host unreachable. But, it is 7682 byte fragmented 
unreachable. And the IP header is a little different.
The IP address of this packet did not make it into the log.

211.205.0.60  45 00 38 00 3C D5 00 00 75 01 2C 85 D3 CD 00 3C 50 A3 B7 BE
03 03 7F 9F 00 00 00 00 45 00 02 02 E0 0C 00 00
And a 14k port unreachable

211.205.0.50  45 C0 1E 02 27 81 00 00 35 01 7F 3D D3 CD 00 32 50 A3 B7 BE
03 01 92 37 00 00 00 00 45 00 02 02 5F 8E 00 00
another 7682 byte host unreach.
The IP address of this packet did not make it into the log.

211.205.0.50  45 C0 1E 02 27 81 00 00 35 01 7F 3D D3 CD 00 32 50 A3 B7 BE
03 01 92 37 00 00 00 00 45 00 02 02 5F 8E 00 00
This one is also not in the log and 7682

211.205.0.66  45 C0 1E 02 F1 F4 00 00 35 01 B4 B9 D3 CD 00 42 50 A3 B7 BE
03 01 92 27 00 00 00 00 45 00 02 02 F7 0F 00 00

211.205.0.58  45 00 38 00 AF 11 00 00 35 01 FA 4A D3 CD 00 3A 50 A3 B7 BE
03 03 7F A1 00 00 00 00 45 00 02 02 A2 40 00 00

this one says 16 bytes extra in the header and 2.5k in size
222.134.119.52  49 03 0A 00 44 00 00 00 80 F9 63 C1 DE 86 77 34 50 A3 B7 BE
C0 A8 97 02 0B 8B 00 19 00 19 02 00 0C 00 00 00

Another 14k - it appears like an ordinary unreach, but length bytes are 
swapped.
211.205.0.60  45 00 38 00 3C D5 00 00 75 01 2C 85 D3 CD 00 3C 50 A3 B7 BE
03 03 7F 9F 00 00 00 00 45 00 02 02 E0 0C 00 00

This one is gigantic
211.205.0.61  45 00 B0 00 2C 89 00 00 75 01 3C 58 D3 CD 00 3D 50 A3 B7 BE
03 03 6C DF 00 00 00 00 45 00 02 02 0B 1F 00 00

and a 14k one.
211.205.0.55  45 00 38 00 64 76 00 00 75 01 04 E9 D3 CD 00 37 50 A3 B7 BE
03 03 7F A4 00 00 00 00 45 00 02 02 0A F6 00 00
---------------------------

At this point I must admit to lack of clue, so unless someone else goes 
'Aha!', my next approach, when the flu I've caught has passed, is to 
find/make a tool to send strange unreachables.

Full memory dump: http://n1.taur.dk/deadmono/freebsd.vmem.gz

/Kasper Pedersen