I may have stumbled upon the issue, but I'm kind of unsure what's going on and hopefully maybe you can help. Disclaimer - everything thus far is virtual, including almost all of the networking. The hypervisor is KVM and all of the adapters throughout are virtio (both sites), that information may prove useful, may not.
So I ganked out the gif tunnel. It's purpose is that it's the simple solution to stretch layer 2 across in order to have a default gateway at the remote site. I'd prefer to not end up with a complicated routing implementation to achieve the desired goal but I can certainly pull the gif tunnel to see if it's the problem. I ended up with something that looks very vanilla:
host1 <-> router3(NAT) <-> wanos1 <-> router1 <-> Internet <-> router2 <-> wanos2 <-> host2
Both router3 and wanos1 are on a network segment with an IPSEC tunnel that reaches wanos2 and host2. Without the gif layer 2 tunnel my problem persisted and looked pretty much the same, host1 cannot rdp to host2. ICMP traffic throughout still flows freely. The peculiar thing I discovered though is that host1 cannot access the https interface on wanos1. Further troubleshooting - I learned that router1 can ssh into wanos1 but router3 cannot. Again, ICMP is doing just fine, and I can tcpdump the interfaces on wanos1 and see ssh traffic targeted to wanos1 but no reply. I can also look at the bridge interfaces on the KVM box to see the same. wanos1 can ssh into router1 but not into router3.
This makes it sound like something is haywire with my virtual network (KVM bridge) so I rebuilt all the interfaces and the bridge, no change. As a point of further troubleshooting I dropped a different machine in where wanos1 was sitting with 2 "physical" interfaces bridged similarly to how wanos is configured and it behaves perfectly.
wanos2 doesn't exhibit the same behavior. The only difference is version 2.6.2 vs 2.6.3, I can't imagine that's the problem. I also built a fresh wanos1 to make sure I didn't foul something up, but the fresh wanos1 behaved just the same.
I'm not sure what could cause TCP to fail while ICMP works perfectly in this configuration. I'm not sure if the problem is in my equipment/configuration or if there is network magic happening within Wanos that I have no knowledge of. I'll keep fiddling, but am hoping someone will spot my problem for me