No optimizing across Layer 2 bridge over IPSEC

[jrdalrymple]

My ultimate goal is to redirect all traffic from a remote site behind a satellite connection over Wanos through my main site. IPSEC already exists and works perfectly, although up to this point I've kept the networks routed across the tunnel. I made some meager attempts at getting routed Wanos functioning, but ultimately felt that was a more complicated solution than just stretching layer 2 across the IPSEC tunnel using OpenBSD gif tunnels - so that's what I did.

In the diagram (attached) pinging from 172.16.104.2 to 172.16.104.1 and vice versa succeeds fine, so Wanos is bridging properly. When I try to move any TCP data however it fails. Wireshark complains of TCP retransmits (attachment) and the upper layer applications don't seem to respond which leads me to believe the data is getting corrupted in flight.

Peer status is often up, so the 2 Wanos boxes can at least find each other.

If I take the Wanos boxes out of the mix everything works perfectly. I've tried both UDP and ipcomp encapsulation. I also of course verified that ipcomp is allowed on my OpenBSD routers. Wanos is configured very vanilla, only change I've made is to the IP so that I can access the boxes.

How can I further troubleshoot? Are my attempts futile? I'd love to be told that I'm missing something obvious.

Thanks

[ahenning]

Hi jr,

My first hunch is that it could possibly be something like an MTU issue due to the IPsec plus GIF encap. One way to quickly test this theory is to change MSS in /tce/etc/wanos/wanos.conf to something small say 1000. If that solves it, then we know we are on the right track.

To apply changes '/tce/etc/wanos/wanos reconfigure' or reset via UI.

[jrdalrymple]

Unfortunately that didn't solve the issue. One additional thing worth noting - when I tcpdump the segment after optimization but before ipsec I'm still seeing what appears to be unoptimized traffic. My testbed has been RDP to an old non-TLS (2003) Windows box thus far, but I've also tried some http and that also fails.

Incidentally, things have also changed in another way that I cannot determine as now I'm not seeing "peer up" either at this stage of the game. ICMP is still working great, and TCP is still making the entire journey to its destination but void of reply, but the "peer up" status no longer happens it seems.

What should I be seeing on the far side of optimization, but before IPSEC?

**edit**

Just one other addition...

is /etc/wanos/clean.sh (or whatever it is) adequate or should I be doing a full reinstall? I have been through a few iterations of configs at this point. Not to be a finger-pointer, but the problem does seem to lie in optimization at this point and I don't want that to be the fault of my own for not properly cleaning up my configs.

[ahenning]

/etc/wanos/clean.sh will reset to default configs.

Ok, it is most unusual, my suggestion would be to setup a lab environment with virtual devices to get an idea of how things are supposed to work before going back into the IPsec+Gif tunnel scenario. The additional gif tunnel also seems redundant but I guess you have valid reasons for adding this.

[jrdalrymple]

I may have stumbled upon the issue, but I'm kind of unsure what's going on and hopefully maybe you can help. Disclaimer - everything thus far is virtual, including almost all of the networking. The hypervisor is KVM and all of the adapters throughout are virtio (both sites), that information may prove useful, may not.

So I ganked out the gif tunnel. It's purpose is that it's the simple solution to stretch layer 2 across in order to have a default gateway at the remote site. I'd prefer to not end up with a complicated routing implementation to achieve the desired goal but I can certainly pull the gif tunnel to see if it's the problem. I ended up with something that looks very vanilla:

host1 <-> router3(NAT) <-> wanos1 <-> router1 <-> Internet <-> router2 <-> wanos2 <-> host2

Both router3 and wanos1 are on a network segment with an IPSEC tunnel that reaches wanos2 and host2. Without the gif layer 2 tunnel my problem persisted and looked pretty much the same, host1 cannot rdp to host2. ICMP traffic throughout still flows freely. The peculiar thing I discovered though is that host1 cannot access the https interface on wanos1. Further troubleshooting - I learned that router1 can ssh into wanos1 but router3 cannot. Again, ICMP is doing just fine, and I can tcpdump the interfaces on wanos1 and see ssh traffic targeted to wanos1 but no reply. I can also look at the bridge interfaces on the KVM box to see the same. wanos1 can ssh into router1 but not into router3.

This makes it sound like something is haywire with my virtual network (KVM bridge) so I rebuilt all the interfaces and the bridge, no change. As a point of further troubleshooting I dropped a different machine in where wanos1 was sitting with 2 "physical" interfaces bridged similarly to how wanos is configured and it behaves perfectly.

wanos2 doesn't exhibit the same behavior. The only difference is version 2.6.2 vs 2.6.3, I can't imagine that's the problem.  I also built a fresh wanos1 to make sure I didn't foul something up, but the fresh wanos1 behaved just the same.

I'm not sure what could cause TCP to fail while ICMP works perfectly in this configuration. I'm not sure if the problem is in my equipment/configuration or if there is network magic happening within Wanos that I have no knowledge of. I'll keep fiddling, but am hoping someone will spot my problem for me

[ahenning]

Hi jr,

I wonder if it could be related to KVM virtual networking, since it is odd that wanos1 can't be accessed from the lan0 side. Unless the VM is not picking up the physical lan0 interface and falling back to single interface router/tunnel mode. Other than that I don't pick up anything in the setup that raises a real flag. 2.6.2 and 2.6.3 has been tested for backwards compatibility, hence that should be ok as well. It wouldn't hurt to update the 2.6.2 to 2.6.3 though.

To check that all the interfaces are detected I would just do a quick 'ifconfig -a', 'ls /sys/class/net/' and if all else fails 'dmesg | grep -i ethernet'. If the real lan0 is there, but it appears that the configured lan0 MAC does not match the intended interface, then 'rm /tce/etc/mactab' to force the interfaces to be rediscovered.

That pings are making it through does seem to indicate that there is some comms. Would it be possible to check on the end host systems if either one is seeing ipcomp traffic when the peers are up? This would indicate that encapsulated traffic is somehow not making into wanos to be decompressed and decapsulated. The end hosts should always see the original traffic. Sometimes it is possible for traffic to flow through the wanos bridge in one direction but not in the opposite direction. What happens is the one wanos detects the peer and starts optimization, but this encapsulated traffic is routed directly to the end stations.

[jrdalrymple]

I wonder if it could be related to KVM virtual networking

Progress.

As the weekend comes to an end I am no longer able to model this in my "production" environment. In starting to build a lab to mimic the entire environment on my laptop I got snagged up pretty darn quick, by the same phenomena that is causing me grief. I made it as far as the attachment depicts and even at that point pinging from Windows to Wanos works, but I can't access the web interface. Again I can see the Windows:##### --> Wanos:443 in tcpdump, but no reply.

With a failure at such a minimalist and green-field setup my only thought was that Wanos doesn't work well with the virtio network adapter so I switched to Realtek. This solves the problem (at least that problem) and the Windows box can then load up Wanos fine. Switching back to virtio breaks it again.

So this may resolve all of my issues going back to the beginning. I'll still finish implementing it in a lab environment to verify that the gif tunnel isn't a problem and if not I'll go back to stuffing it into my live environment. The question I have to ask now is why doesn't the virtio driver work properly? It would be super cool if that could get looked at, I hate emulating anything in software on my hypervisor machines, for obvious reasons.

Thanks!

[ahenning]

Interesting, actually Lew emailed me about this thread suggesting that the problem could be due to the virtio/realtek/rtl8139 issue which he picked up while writing up the KVM documentation. I did not put two and two together quick enough.

We can take a look into the driver issue, but we are a day or two away from launching an Ubuntu version. We are in the process of testing all the hypervisors and will ensure everything is ok on KVM as well. Might be worth the time to implement v.3.0 rather.

[jrdalrymple]

Realtek adapter fixed it in both the lab and production environments. I did still have to do some MSS tuning, end game though is that for now in a KVM environment avoid virtio for your Wanos network adapters.

OpenBSD gif interfaces are fully cooperative.

Thanks for your help!