Author Topic: Strange delay through wan interface - KVM  (Read 7809 times)

lexore

  • Member
  • ***
  • Posts: 6
    • View Profile
Strange delay through wan interface - KVM
« on: March 03, 2015, 09:37:26 PM »
Hello.
I have a problem with wanos.
Could you please give an advice?

I have a configuration with wanos VM and proxy VM on virtualization server.
Network looks like this:
HOST -<bridge>- (lan) WANOS (wan) -<bridge>- PROXY

Sometimes ping from WANOS to PROXY through the wan interface looks like this:
Code: [Select]
tc@wanos:~$ ping 10.250.254.3
PING 10.250.254.3 (10.250.254.3): 56 data bytes
64 bytes from 10.250.254.3: seq=18 ttl=64 time=21004.807 ms
64 bytes from 10.250.254.3: seq=19 ttl=64 time=20004.707 ms
64 bytes from 10.250.254.3: seq=20 ttl=64 time=19003.630 ms
64 bytes from 10.250.254.3: seq=21 ttl=64 time=18003.504 ms
64 bytes from 10.250.254.3: seq=22 ttl=64 time=17003.300 ms
64 bytes from 10.250.254.3: seq=23 ttl=64 time=16003.196 ms
64 bytes from 10.250.254.3: seq=24 ttl=64 time=15003.083 ms
64 bytes from 10.250.254.3: seq=25 ttl=64 time=14002.993 ms
64 bytes from 10.250.254.3: seq=26 ttl=64 time=13002.876 ms
64 bytes from 10.250.254.3: seq=27 ttl=64 time=12002.761 ms
64 bytes from 10.250.254.3: seq=28 ttl=64 time=11002.689 ms
64 bytes from 10.250.254.3: seq=29 ttl=64 time=10002.627 ms
64 bytes from 10.250.254.3: seq=30 ttl=64 time=9002.477 ms
64 bytes from 10.250.254.3: seq=31 ttl=64 time=8002.272 ms
64 bytes from 10.250.254.3: seq=32 ttl=64 time=7002.107 ms
64 bytes from 10.250.254.3: seq=33 ttl=64 time=6001.997 ms
64 bytes from 10.250.254.3: seq=34 ttl=64 time=5001.834 ms
64 bytes from 10.250.254.3: seq=35 ttl=64 time=4001.695 ms
64 bytes from 10.250.254.3: seq=36 ttl=64 time=3001.530 ms
64 bytes from 10.250.254.3: seq=37 ttl=64 time=2001.370 ms
64 bytes from 10.250.254.3: seq=38 ttl=64 time=1001.267 ms
64 bytes from 10.250.254.3: seq=40 ttl=64 time=28009.804 ms
64 bytes from 10.250.254.3: seq=41 ttl=64 time=27009.673 ms
64 bytes from 10.250.254.3: seq=42 ttl=64 time=26009.510 ms
64 bytes from 10.250.254.3: seq=43 ttl=64 time=25009.394 ms
64 bytes from 10.250.254.3: seq=44 ttl=64 time=24009.448 ms
64 bytes from 10.250.254.3: seq=45 ttl=64 time=23009.321 ms
64 bytes from 10.250.254.3: seq=46 ttl=64 time=22009.173 ms
64 bytes from 10.250.254.3: seq=47 ttl=64 time=21009.041 ms
64 bytes from 10.250.254.3: seq=48 ttl=64 time=20008.946 ms
64 bytes from 10.250.254.3: seq=49 ttl=64 time=19008.809 ms
64 bytes from 10.250.254.3: seq=50 ttl=64 time=18008.672 ms
64 bytes from 10.250.254.3: seq=51 ttl=64 time=17008.527 ms
64 bytes from 10.250.254.3: seq=52 ttl=64 time=16008.402 ms
64 bytes from 10.250.254.3: seq=53 ttl=64 time=15008.245 ms
64 bytes from 10.250.254.3: seq=54 ttl=64 time=14008.103 ms
64 bytes from 10.250.254.3: seq=55 ttl=64 time=13008.033 ms
64 bytes from 10.250.254.3: seq=68 ttl=64 time=31005.465 ms
64 bytes from 10.250.254.3: seq=69 ttl=64 time=30005.320 ms
64 bytes from 10.250.254.3: seq=70 ttl=64 time=29005.229 ms
64 bytes from 10.250.254.3: seq=71 ttl=64 time=28005.096 ms
64 bytes from 10.250.254.3: seq=72 ttl=64 time=27004.903 ms
64 bytes from 10.250.254.3: seq=73 ttl=64 time=26004.760 ms
64 bytes from 10.250.254.3: seq=74 ttl=64 time=25004.613 ms
64 bytes from 10.250.254.3: seq=75 ttl=64 time=24004.519 ms
64 bytes from 10.250.254.3: seq=76 ttl=64 time=23004.373 ms
64 bytes from 10.250.254.3: seq=77 ttl=64 time=22004.235 ms
64 bytes from 10.250.254.3: seq=78 ttl=64 time=21004.128 ms
64 bytes from 10.250.254.3: seq=79 ttl=64 time=20004.005 ms
64 bytes from 10.250.254.3: seq=80 ttl=64 time=19003.870 ms
64 bytes from 10.250.254.3: seq=81 ttl=64 time=18003.674 ms
64 bytes from 10.250.254.3: seq=82 ttl=64 time=17003.534 ms

In the same time ping from WANOS to HOST through lan looks fine:
Code: [Select]
tc@wanos:~$ ping 10.250.254.1
PING 10.250.254.1 (10.250.254.1): 56 data bytes
64 bytes from 10.250.254.1: seq=0 ttl=64 time=0.522 ms
64 bytes from 10.250.254.1: seq=1 ttl=64 time=0.451 ms
64 bytes from 10.250.254.1: seq=2 ttl=64 time=0.334 ms
64 bytes from 10.250.254.1: seq=3 ttl=64 time=0.380 ms
64 bytes from 10.250.254.1: seq=4 ttl=64 time=0.417 ms
64 bytes from 10.250.254.1: seq=5 ttl=64 time=0.532 ms
64 bytes from 10.250.254.1: seq=6 ttl=64 time=0.526 ms
64 bytes from 10.250.254.1: seq=7 ttl=64 time=0.534 ms
64 bytes from 10.250.254.1: seq=8 ttl=64 time=0.300 ms
64 bytes from 10.250.254.1: seq=9 ttl=64 time=0.308 ms

I can see very accurate delay (30 seconds, 29, 28, etc) between WANOS and PROXY.
It's very strange, because they connected directly.
They are don't have heavy load, and there are no traffic between them, only ping.
I noticed that this problem only through wan interface.
Maybe WANOS generate delay in some circumstances?


details:
HOST OS: ubuntu 14.04
HOST VM type: libvirt+qemu-kvm
WANOS ver: 1.4.1
PROXY OS: ubuntu 14.04
bridge stp: off (i tried on and off)

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #1 on: March 03, 2015, 10:29:22 PM »
That most certainly looks like a loop of some sort. ICMP traffic is bypassed completely.


Besides this, once Wanos optimizes and encapsulates the traffic, the proxy will not receive the traffic destined to it. Should the proxy not be on the host side?

Do you have the steps that you used to load Wanos in KVM and I can try to simulate the scenario?
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

lexore

  • Member
  • ***
  • Posts: 6
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #2 on: March 08, 2015, 09:27:35 PM »
Oh, i'm very sorry, my mistake, i mixed up interface names.
Big delay on lan interface, not wan.
If you interesting, here is the scheme in complete form:
http://scr.lexore.net/20150305-y1h-53kb.jpg
We have two virtualization servers with two VM on each: proxy and wanos.
Wanos servers placed between proxies for traffic compression.

When this schema run in production, after some time i see very big delay between wanos and proxy.
As i noted, this is on lan interface of wanos.
After reboot delay dissapear.
For example, in previous post i showed delay between wanos (10.250.254.2) and proxy (10.250.254.3).
After reboot delay is gone:
Code: [Select]
tc@wanos:~$ ping 10.250.254.3
PING 10.250.254.3 (10.250.254.3): 56 data bytes
64 bytes from 10.250.254.3: seq=1 ttl=64 time=0.857 ms
64 bytes from 10.250.254.3: seq=2 ttl=64 time=0.592 ms
64 bytes from 10.250.254.3: seq=3 ttl=64 time=0.644 ms
64 bytes from 10.250.254.3: seq=4 ttl=64 time=0.656 ms
64 bytes from 10.250.254.3: seq=5 ttl=64 time=0.635 ms

So, this delay appear after some time of work, when http traffic starts.
I saw this situation on both pairs of servers (proxy1-wanos1 and proxy2-wanos2) - big delay on lan interface after some time of work.
I don't think, that this is loop (if you about traffic) because after reboot all is fine.

About your question to simulate i'll try to explain.
But first i want to say, that i'm ready to provide access to wanos servers if you interested or if will be problems with reproduce steps.

On virtualization server i use package "libvirt-bin" for manage VMs.
I used GUI virt-manager to create and configure VMs.
But you can use cli command virsh and xml files (see below).

I created VM with 2 nics (driver pcnet) for wanos.
For wanos hdd i convert vmdk image to raw format.
I attached xml with this VM config (wanos-msk1.xml).
You can create VM with command:
Code: [Select]
virsh create wanos-msk1.xmlVm for wanos has this settings:
mem: 2GB
cpu: 4
hdd: /var/lib/libvirt/images/wanos-msk1.raw
hdd format: raw
hdd size: 64 GB
lan interface connects to bridge: br1
wan interface connects to bridge: host2wanos
VNC session for this VM available on port 5900.

And i created VM with 2 nics for proxy, and install ubuntu to it.
I also attached xml for it (proxy-msk1.xml)
You can create VM with command:
Code: [Select]
virsh create proxy-msk1.xmlThis VM has this settings:
mem: 2GB
cpu: 4
hdd: /var/lib/libvirt/images/proxy-msk1.raw
hdd format: raw
hdd size: 10GB
interface to wanos connects to bridge: br1
interface to inet connects to bridge: br0
VNC session for this VM available on port 5901.

Before start VMs, you need to create three bridges:
* To connect proxy with real network (in my configs it has name br0)
If you virtualization server has nic eth0, commands will be:
Code: [Select]
brctl addbr br0
brctl addif br0 eth0
ip a del <ip/mask> dev eth0
ip a add <ip/mask> dev br0
ip ro add default via <dgw> #default gateway can gone after delete ip

* to connect proxy and wanos (in my configs it has name br1)
Code: [Select]
brctl addbr br1
* to connect wanos with real word
in my configs libvirt creates additional inteface on virtualization server.
You can create it with command:
Code: [Select]
virsh net-create <host2wanos.xml>
When network ready, you can start VM's with commands:
Code: [Select]
virsh start proxy-msk1
virsh start wanos-msk1

libvirt will create virtual nics (vnet*) and connect they to bridges.

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #3 on: March 08, 2015, 10:34:15 PM »
Hi,

Thanks for the very detailed info. It helps to take a good look at this, because we would like to build up more info on how to make Wanos work seamlessly in KVM.

Some questions:
The USB/HDD image is a RAW disk image of the same VHD/VMDK file. Will it work or is it required to use "qemu-img create" to make the RAW image compatible with KVM?

Is this thread possibly of any help: Wanos throughput on KVM
Especially the virtio-net-pci nic drivers part.

The reboot that solves the latency issue makes me more suspect a loop of some sort (Since traffic load is already ruled out). But agree it could be something else and I'm willing to have an open mind and look at other options.

Is it possible to not reboot Wanos, but simply disable the bridge on the host server side for a minute and see if the latency continues after enabling the bridge again?

And a similar test, without rebooting, simply "ifconfig wan0/lan0 down" and checking the latency after enabling them again. Also same test, just reset the wanos service from the GUI. Maybe you have tried this already?

Perhaps the Interface graphs show a trend of increased traffic that is not supposed to be there?

v2.0.3 has Routed mode available in Express. It might help to use less Linux/KVM bridges.

Are there any custom configs or modules used in the proxy? We are planning to make the web cache proxy available in Express soon as well.


I will definitely take a look at the KVM setup, since it would be very good to support it and get more users to help find solutions for these type of issues. Unfortunately 'free' time is tight at the moment and it will probably only be after tunnel mode and the amazon AWS image is complete.




« Last Edit: March 10, 2015, 12:17:34 AM by ahenning »
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

peter

  • Contributor
  • Full Member
  • *****
  • Posts: 51
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #4 on: March 11, 2015, 12:23:55 AM »
     I had similar problem when trying to put wanos on kvm.  The throughput is not stable.  Sometimes it was good and sometimes it was bad.  Initially, I thought I had caught problem regarding the physical NIC and its driver. But, later I found that the "problem nic" worked fine with vmware and it is very stable.  I had no chice but to switch vmware.  But, I still hope wanos would work on KVM because open source matters.

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #5 on: March 11, 2015, 02:35:44 AM »
Thanks Peter,

I still suspect the bridging in KVM. Maybe with the Router Mode in v2.0.3 its worth a look again? Router mode only needs one interface. Is it possible to route traffic from the host to the guest vm?
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

peter

  • Contributor
  • Full Member
  • *****
  • Posts: 51
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #6 on: March 11, 2015, 03:56:38 AM »
one thing I forgot to mention, I tried wanos on open source XEN as well.  But, the performance was not stable as well. 

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #7 on: March 11, 2015, 09:37:53 AM »
Does these hypervisors support Vlans for true isolation? For example like we use on ESXi?

If not, I think Router mode is probably going to be the most reliable way to channel traffic through the guest vm.
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs

lexore

  • Member
  • ***
  • Posts: 6
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #8 on: April 06, 2015, 08:47:46 PM »
Sorry for the long answer.
I found solution, that works for me!
I changed driver in KVM from "pcnet" to "e1000" on VM options, for both NICs.
With "e1000" ping works absolutely perfect, with no delays in 24 hours (i keep ping working for 24 hours).
It was with http load test, so i think, problem solved.
I'll remind, it's 1.4.1 version.

I also tried driver "virtio", it's special driver for virtualized NICs.
With this driver ping works, but http answers didn't pass through wanos-wanos.
virtio is a specific driver, it's not emulate many physical things, like carrier speed and so on.
It's something like standart driver for VM in KVM/xen.
So, if you will had a free time, i recommend try to make this driver working.

Thanks for the link "Wanos throughput on KVM", i missed it when first read your answer.

ahenning

  • Team Wanos
  • Administrator
  • Full Member
  • *****
  • Posts: 629
    • View Profile
Re: Strange delay through wan interface - KVM
« Reply #9 on: April 07, 2015, 12:36:33 AM »
Hi Lexore,

Thanks for the great feedback. Good to know to use the e1000 NIC option on KVM.
CCIE RS, CCIE SP, Mnet&sys

Note: Forum posts may be outdated. Please see the latest documentation at wanos.co/docs