Help Needed Converting KVM network Non-VLAN network to VLANs, odd

Mon Jun 22 05:39:49 UTC 2015

Hi!

I apologize if this is not something I should have posted here, but I've
come to value the insights and experience of the people on this list a lot,
and I am hoping my problem isn't unique. I am also sorry for the long read.

I have been to the forums of the devices in play in this problem, and while
Red Hat has been a huge help, they all hand off when they hear about the
other devices in play.

Some background:
I have a Sophos UTM ASG220 serving as gateway device for a number of
networks, with a Cisco 2960 network switch, and a raft  of Red Hat 6.6
servers running KVM and hosting multiple guests, with the guests being on
different network subnets. The UTM has its LAN interface populated  with
multiple virtual interfaces (its really a stripped down, optimized
RHEL-type Linux machine under the hood) as gateways for all the network
subnets except for the primary network it was created with during
installation. I have VLANs defined on the switch, and the KVM hosts are
having bonded interfaces (mode 1, based on RHN support advice), VLAN sub
interfaces and bridges configured for each network, and each guest is
attached to its appropriate bridge and 8021q is setup. Without involving
the UTM, VLAN traffic transverses beautifully, between swich, KVM hosts and
guests, I have no issues there

That said, this is what is happening:
I am successful in generating new VLAN interfaces on the Sophos UTM (but
with a different IP address) to replace the existing gateway virtual IP
address (for instance, for test network, virtual interface gateway address
is 10.11.0.253, and the VLAN interface to replace it is 10.11.0.253). At
first instance the guests and the kvm host are able to ping the switch, the
newVLAN gateway interface and the old virtual gateway interface, after the
VLAN is in place. But if I try to remove the old virtual interface (eg
10.11.0.253), then networking starts acting weird. The switch VLAN address
(say 10.11.0.7) isunable toping or reach the guests (say 10.11.0.36) on the
VLAN, but it can reach the kvm host vlan bridge (say 10.11.0.4) address,
and it can reach the Sophos gateway (10.11.0.254,VLAN address). Even after
bring the gateway virtual interface (10.11.0.253) back up the situation
remainsfor a while. The guests can reach each other on the same VLAN, but
cannot ping the switch VLAN interface address, and cannot ping their VLAN
gateway address, or route traffic to other external networks). But the
guests can reach the LAN DNS servers, which are ona different subnet
entirely (192.168.2.0)! But theguests also can only reach the DNS servers
on the 192.168.2.0 subnet, they cannot reach all the addresses. Arping
responds to and from all network machines/devices while all this is going
on. This continued for a while even after rebooting the switch, and
bringing up and down the gateway network interfaces. Then suddenly things
started working again (but with the gateway virtual and VLAN addresses both
up).I am successful in generating anew VLAN interface (but with a different
IP address) to replace the existing gateway virtual IP address (for
instance, for test network, virtual interface gateway address is
10.11.0.253, and the VLAN interface to replace it is 10.11.0.253). At first
instance the guests and the kvm host are able to ping the switch, the
newVLAN gateway interface and the old virtual gateway interface, after the
VLAN is in place. But if I try to remove the old virtual interface (eg
10.11.0.253), then networking starts acting weird. The switch VLAN address
(say 10.11.0.7) isunable toping or reach the guests (say 10.11.0.36) on the
VLAN, but it can reach the kvm host vlan bridge (say 10.11.0.4) address,
and it can reach the gateway (10.11.0.254,VLAN address). Even after
bringing  the gateway virtual interface (10.11.0.253) back up the situation
remains for a while. The guests can reach each other on the same VLAN, but
cannot ping the switch VLAN interface address, and cannot ping their VLAN
gateway address, or route traffic to other external networks). But the
guests can reach the LAN DNS servers, which are on a different subnet
entirely (192.168.2.0)! The guests also can only reach the DNS servers on
the 192.168.2.0 subnet, they cannot reach all the addresses.

 Arping responds to and from all network machines/devices while all this is
going on.
 This continued for a while even after clearing the arp-caches,  rebooting
the switch, and bringing up and down the gateway network interfaces.

Then suddenly things started working again (but with the gateway virtual
and VLAN addresses both up).

I'd love some insight to what's happening and how I can fix this.