Author Topic: Peers down - IPSec VPN  (Read 9058 times)

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Peers down - IPSec VPN
« on: October 14, 2014, 10:02:07 AM »
Hello!
I have network diagram the same as above, I write optimization rule on edge (Branch LAN -> 0.0.0.0/0) and core (0.0.0.0/0 -> Branch LAN), but I don't see IPComp packet from edge and traffic is broken. I used Wireshark and saw a lot of TCP Retransmission packet. I can't use Multisite because I have a lot of branch (greater than 100), but don't have much wanos boxes. On lab all work fine, I build network the same as my production network, but without IPSec between branch and HQ. In production network I use Cisco VTI for connecting spoke to hub.

As well, I have nanX on WAN to LAN optimization Ratio graph.

Do you have any idea?






Admin edit: Changed image format from tiff to jpg
« Last Edit: October 14, 2014, 11:35:59 AM by ahenning »

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #1 on: October 14, 2014, 11:38:33 AM »
Hi Misha,

The rules look fine if the optimization is only supposed to work for the one 10.90.250/0 site. Are the peers up in the 'peer status' tab?
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #2 on: October 14, 2014, 11:52:45 AM »
Thank you for reply!
After change policy I saw "Debug : Peer Alive", but traffic between networks was corrupted. For 5-10 minutes peer status change to "down". My edge device is HDD appliance, my core is VM on VMWare, but they has the same version.

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #3 on: October 14, 2014, 12:47:16 PM »
Very strange problem. Is bypass traffic e.g. HTTPS affected in any way?
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #4 on: October 14, 2014, 05:18:00 PM »
I think so. Yes, ssh affected any way. I think problem depends with IPSec or MTU. Tomorrow I'm going create simply GRE tunnel and check this one.

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #5 on: October 15, 2014, 04:06:56 PM »
Hello again! :-)
New day and I have new question. I make IPsec on my lab network, the same as my production network. Please, see attach. On lab network all work fine!

Config for branch router:
Code: [Select]
Current configuration : 2015 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname br2
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$0sZ8$JIA80Csa9sJxMGeFTS99H/
!
no aaa new-model
tdm clock E1 1/0 both export line
voice-card 1
!
ip cef
!
!
!
!         
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username cisco privilege 15 secret 5 $1$nG08$dAIyTE/GmPyaZFRwa.fa8.
!
!
controller E1 1/0
!
!         
crypto keyring MyKEY
  pre-shared-key address 0.0.0.0 0.0.0.0 key MegaKey
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 120 20
crypto isakmp nat keepalive 20
crypto isakmp profile MyPROFILE
   keyring MyKEY
   match identity address 0.0.0.0
!
crypto ipsec security-association lifetime seconds 43200
!
crypto ipsec transform-set MySET esp-3des esp-sha-hmac
!
crypto ipsec profile MyPROFILE
 set transform-set MySET
!
!
!         
!
!
interface Tunnel10
 description --- To MSK ---
 bandwidth 2048
 ip unnumbered FastEthernet0/0.21
 tunnel source FastEthernet0/0.20
 tunnel destination 192.168.1.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile MyPROFILE
!
interface FastEthernet0/0
 no ip address
 speed auto
!
interface FastEthernet0/0.20
 encapsulation dot1Q 20
 ip address 192.168.1.6 255.255.255.252
!
interface FastEthernet0/0.21
 encapsulation dot1Q 21
 ip address 10.1.2.1 255.255.255.0
!         
interface Serial0/0
 no ip address
 shutdown
 no fair-queue
!
router ospf 10
 log-adjacency-changes
 passive-interface default
 no passive-interface FastEthernet0/0.20
 network 192.168.1.4 0.0.0.3 area 0
!
router rip
 version 2
 passive-interface default
 no passive-interface Tunnel10
 network 10.0.0.0
 no auto-summary
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
ip prefix-list RIP seq 5 permit 10.1.2.0/24
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
line con 0
line aux 0
line vty 0 4
 login local
!
end

And config for HQ router
Code: [Select]
Current configuration : 2503 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname br1
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$0sZ8$JIA80Csa9sJxMGeFTS99H/
!
no aaa new-model
ip cef
!
!
!
!
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!         
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username cisco privilege 15 secret 5 $1$nG08$dAIyTE/GmPyaZFRwa.fa8.
!
!
!
crypto keyring MyKEY
  pre-shared-key address 0.0.0.0 0.0.0.0 key MegaKey
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 120 20
crypto isakmp nat keepalive 20
crypto isakmp profile MyPROFILE
   keyring MyKEY
   match identity address 0.0.0.0
   virtual-template 1
!
!
crypto ipsec transform-set MySET esp-3des esp-sha-hmac
!
crypto ipsec profile MyPROFILE
 set transform-set MySET
!
!
!
!
!
interface Loopback0
 description --- Router ID ---
 ip address 10.1.100.1 255.255.255.255
!
interface Ethernet0/0
 no ip address
 shutdown
 half-duplex
!
interface FastEthernet0/0
 no ip address
 speed auto
!
interface FastEthernet0/0.10
 encapsulation dot1Q 10
 ip address 192.168.1.2 255.255.255.252
!
interface FastEthernet0/0.11
 encapsulation dot1Q 11
 ip address 10.1.1.1 255.255.255.0
!
interface Virtual-Template1 type tunnel
 description --- To Branch ---
 ip unnumbered Loopback0
 tunnel source FastEthernet0/0.10
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile MyPROFILE
!
router ospf 10
 log-adjacency-changes
 redistribute static
 passive-interface default
 no passive-interface FastEthernet0/0.10
 network 192.168.1.0 0.0.0.3 area 0
!
router ospf 100
 log-adjacency-changes
 redistribute rip subnets route-map RIP->OSPF
 passive-interface default
 no passive-interface FastEthernet0/0.11
 network 10.1.1.0 0.0.0.255 area 0
 network 10.1.100.1 0.0.0.0 area 0
!
router rip
 version 2
 passive-interface FastEthernet0/0.10
 passive-interface FastEthernet0/0.11
 passive-interface Loopback0
 network 10.0.0.0
 default-information originate
 distribute-list prefix DEF out
 no auto-summary
!
ip forward-protocol nd
ip route 10.100.1.0 255.255.255.0 10.1.1.2
!
!
no ip http server
no ip http secure-server
!
!
ip prefix-list DEF seq 5 permit 0.0.0.0/0
!
ip prefix-list Mypfl seq 1 permit 10.1.0.0/16 le 24
!
route-map RIP->OSPF permit 10
 match ip address prefix-list Mypfl
 set tag 5090
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
line con 0
line aux 0
line vty 0 4
 login local
!
end

What difference between production and lab network? Nothing exept Internet for transport and in lab network wanos core and edge still on VMWare.
I dumped traffic from WAN interface for edge and core in production network and I saw IPComp between source and destination hosts, but on LAN interface I saw only TCP Retransmit from source to destination. I'm sad. I know what I have mistake, but where?

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #6 on: October 15, 2014, 06:51:14 PM »
Yes, I agree, the only difference is in the Internet for transport between sites. This suggests to me that maybe the problem is MTU related. Is it possible that the Internet connected routers have extra encapsulation like PPPoE or another VPN header?

One thing that I am not sure of is that in the diagram and router configs they establish the VPN over the 192.168.1.x private addresses. I understand this will work in the lab, but if this is the production config as well, then it seems this ipsec tunnel is running over a second tunnel (e.g. Another ipsec tunnel or gre/l2tp). In this scenario the extra encapsulation can definitely contribute to MTU problems. One ipsec encapsulation would be fine though.

The step 1 would be to bypass all traffic. Doing this will pass through all traffic untouched. All traffic should work across the VPN. If not, then there is still some configuration missing for the VPN. Once the two sites can communicate without any issues without Wanos, then we know it is ready to start with the wanop configs.

As a last resort you can try UDP Encapsulation, which is enabled in /tce/etc/wanos/wanos.conf UDPENCAP=Enable
This is required on both sides and Configure > Reset to apply the change. If there are MTU issues, the routers should fragment the packets and Wanos will try to reassemble them. This is not ideal and not recommended as the final solution, but can help in the troubleshooting process.
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #7 on: October 16, 2014, 04:24:41 AM »
Yes, first I check MTU and MSS, it's okay. On prod. network pass through rules work fine, all traffic work across the VPN. But when I write rule for networks, and I saw
Code: [Select]
"Debug : Peer Alive", traffic has been corrupted.

Now, some magic! Thank you, I enabled UDP encapsulation and network still alive. I saw on core and edges:

Code: [Select]
"Debug : Peer Alive
Debug : Datastore Sync Update"

Why? I'm think because TCP has SEQ number, but why this one was changed?

Sometimes, I'm see messages on core:

Code: [Select]
"expensive Packet::put; have 1912 wanted 2856
ToDevice(lan0): Message too long"

It's normal?

Thank you very much.
« Last Edit: October 16, 2014, 08:26:45 AM by Misha »

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #8 on: October 16, 2014, 11:30:36 AM »
Hi Misha,

The datastore sync happens if there was some packetloss on the network previously. It could have been due to the previous problems, so it would be a good idea to clear the datastore and start off fresh.

Clever design for branch type one. The VPN traffic should pass-through and should not affect anything. Are the problems affected by both designs or only this internet based branch type?

Since the UDPEncap works, it can mean a few things:
WAN is dropping proto 108 packets.
WAN can't NAT/PAT proto 108 packets.
WAN can't fragment proto 108 packets.

If the WAN is not configured to drop proto 108 and there is no NAT configured as in the lab configs, then the likely one is the fragment problem due to MTU.

The expensive packet warning is not normal, we should investigate why this is happening. Could you please run ethtool -k lan0 and ethtool -k wan0?
'ethtool -i' would also be useful.
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #9 on: October 16, 2014, 12:02:34 PM »
Hi!
The problems affected by both designs.
I changed MTU to 512 on workstation, but have problem. I think, packets corrupted into edge device, because I'm seeing proto108 packets between core and edge, core and server, but seeing nothing from edge to workstation

The edge (over Internet):
Code: [Select]
tc@edge:~$ ethtool -k lan0
Offload parameters for lan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off
tc@edge:~$ ethtool -k wan0
Offload parameters for wan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

tc@edge:~$ ethtool -i wan0
driver: e1000e
version: 1.3.10-k2
firmware-version: 0.13-4
bus-info: 0000:00:19.0
tc@edge:~$ ethtool -i lan0
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version: N/A
bus-info: 0000:01:00.0

The edge (over IP VPN)
Code: [Select]
tc@mo-domodedovo:~$ ethtool -k wan0
Offload parameters for wan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
tc@mo-domodedovo:~$ ethtool -k lan0
Offload parameters for lan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

tc@mo-domodedovo:~$ ethtool -i wan0
driver: r8169
version: 2.3LK-NAPI
firmware-version: rtl_nic/rtl8168d-1.fw
bus-info: 0000:01:00.0
tc@mo-domodedovo:~$ ethtool -i lan0
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version: N/A
bus-info: 0000:03:02.0
tc@mo-domodedovo:~$

and the core:
Code: [Select]
tc@core:~$  ethtool -k lan0
Offload parameters for lan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
tc@core:~$  ethtool -k wan0
Offload parameters for wan0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

tc@core:~$  ethtool -i wan0
driver: vmxnet3
version: 1.1.18.0-k-NAPI
firmware-version: N/A
bus-info: 0000:03:00.0
tc@core:~$  ethtool -i lan0
driver: vmxnet3
version: 1.1.18.0-k-NAPI
firmware-version: N/A
bus-info: 0000:0b:00.0

Thank you.

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #10 on: October 16, 2014, 12:18:57 PM »
Ok, thanks for the details. They are all correct, so then the expensive packet message must be due to the fragment reassembling. I take it the expensive packet message was not present previously and only started when UDPEncap was enabled?

Also are all the Wanos interfaces on the standard MTU 1500?

The proto 108 messages should only be visible between wanos devices e.g:
PC --- TCP --- Wanos --- Proto108 --- Wanos --- TCP --- Server
« Last Edit: October 16, 2014, 12:24:11 PM by ahenning »
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

Misha

  • V.I.P
  • Member
  • ***
  • Posts: 7
    • View Profile
Re: Peers down - IPSec VPN
« Reply #11 on: October 16, 2014, 04:44:43 PM »
Hello!

Ok, thanks for the details. They are all correct, so then the expensive packet message must be due to the fragment reassembling. I take it the expensive packet message was not present previously and only started when UDPEncap was enabled?

Also are all the Wanos interfaces on the standard MTU 1500?
Yes, this message only started when UDPEncap was enabled.
Yes, all the Wanos interfaces are on the standard MTU 1500. When I used TCP encapsulation, I have been changed MTU to 1000, but packets between edge and workstation are damaged any way. I saw that peer on edge has been down until peer on core has been up.

The proto 108 messages should only be visible between wanos devices e.g:
PC --- TCP --- Wanos --- Proto108 --- Wanos --- TCP --- Server

Yes, I'm understand :-)
« Last Edit: October 17, 2014, 05:02:47 AM by Misha »

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Peers down - IPSec VPN
« Reply #12 on: December 03, 2014, 11:25:07 PM »
Moved to a separate topic.

Findings: Newer Cisco hardware/IOS drops proto 108 traffic over IPsec GRE tunnels. Feature or bug... Current workaround is to use a different encapsulation e.g. GRE only on the Cisco VPN or enable udpencap on wanos.
« Last Edit: December 11, 2014, 02:37:58 PM by ahenning »
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs