Hi,

> racoon: INFO: respond new phase 2 negotiation: 
> m0n0.ip[500]<=>remote.ip[500]
> Pluto[119] #180: max number of retransmissions (2) reached STATE_QUICK_I1

It can be the following scenario:

1. "Pluto" is transmitting info to m0n0wall.

2. m0n0wall receives fragmented packets (incoming udp on port 500) but
default rule for IKE packets is dropping fragments.
> # Pass IKE packets
> pass in quick on {$ifname} proto udp from any to {$ip} port = 500

So, m0n0wall does not get expected proposal, "Pluto" transmit it again
and again until max number of allowed retransmissions is reached.

If this is the case then you should see dropped udp packets (no port 
reported)
in your m0n0wall firewall log.
And read thread "Problem with IPSec VPN Tunnel - MTU-Size?". There is
a solution offered that works for me.

regards,
Marek