_files/genppageinit.gif){kind=small}
_files/ibm-comm-wordmark.png){kind=small}
_files/blank.gif){kind=spacer}
_files/iconWarning.gif "IC Sunset")
The
developerWorks Connections platform will be sunset on December 31,
2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.
IVM and VLAN TaggingI was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.
As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.
The Blade server
name was Serv
# lsdev -Cc adapter ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Logical Host Ethernet Port (lp-hea) ent2 Available Virtual I/O Ethernet Adapter (l-lan) ent3 Available Virtual I/O Ethernet Adapter (l-lan) ent4 Available Virtual I/O Ethernet Adapter (l-lan) ent5 Available Virtual I/O Ethernet Adapter (l-lan) ent6 Available EtherChannel / IEEE 802.3ad Link Aggregation ent7 Available Shared Ethernet Adapter ent9 Available Virtual I/O Ethernet Adapter (l-lan) fcs0 Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601) fcs1 Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601) ibmvmc0 Available Virtual Management Channel lhea0 Available Logical Host Ethernet Adapter (l-hea) sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter usbhc0 Available 00-08 USB Host Controller (33103500) usbhc1 Available 00-09 USB Host Controller (33103500) usbhc2 Available 00-0a USB Enhanced Host Controller (3310e000) vhost0 Available Virtual SCSI Server Adapter vsa0 Available LPAR Virtual Serial Adapter vts0 Available Virtual TTY Server Adapter
The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.
# lsattr -El ent7 accounting
enabled Enable per-client accounting of network
stat ctl_chan
Control Channel
adapter for SEA fail gvrp
no Enable GARP VLAN Registration Protocol
(GVR ha_mode
disabled High Availability
Mode jumbo_frames
no Enable Gigabit Ethernet Jumbo
Fram large_receive
no Enable receive TCP segment
aggr largesend
0 Enable Hardware Transmit TCP
Rese netaddr
0 Address to
ping pvid
68 PVID to use for the SEA
devi pvid_adapter ent9 Default virtual adapter to use for non-VLAN-tagged packets True qos_mode
disabled
N/A real_adapter
ent6 Physical adapter associated with the
SEA
thread
1 Thread mode enabled (1) or disabled
(0) virt_adapters ent9 List of virtual adapters associated with the SEA (comma separated) True
$ entstat -all ent8 | grep -i vlan VLAN Ids : VLAN Extract: False VLAN tagged filtering mode: Filter according to VLAN permit array Max number of VLAN IDs per HEA port: 20 VLAN Extract: False VLAN tagged filtering mode: Filter according to VLAN permit array Max number of VLAN IDs per HEA port: 20 Invalid VLAN ID Packets: 5388 Port VLAN ID: 68 VLAN Tag IDs: None
On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.
So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.
We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.
$ lshwres -r virtualio --rsubtype eth --level lpar lpar lpar lpar lpar lpar lpar
We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.
First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.
$
chhwres -m Serv $
chhwres -m Serv
$ lshwres -r virtualio --rsubtype eth --level lpar lpar lpar lpar lpar lpar lpar
Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.
$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11 ent7 Available en7 et7
$ entstat -all ent7 | grep -i vlan VLAN Ids : VLAN Extract: False VLAN tagged filtering mode: Filter according to VLAN permit array Max number of VLAN IDs per HEA port: 20 VLAN Extract: False VLAN tagged filtering mode: Filter according to VLAN permit array Max number of VLAN IDs per HEA port: 20 Invalid VLAN ID Packets: 5388 Port VLAN ID: 11 VLAN Tag IDs: 68
Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.
A related link: http
|
_files/photo_003.png){kind=small}
Excelent guide for VLANs. Does anybody have a similar guide for non-VLAN tagging?
_files/photo_006.png){kind=small}
Hello Chris, I have a similar issue on a FLEX p260 node with VIOS and AIX 6.1 LPARs. I do not have FSM nor HMC. I am using IVM. The VIO has to be in VLAN 250 and the LPARs in VLAN 2. I could put the VIO on VLAN 250 but the AIX LPARS in VLAN 2 are not able to communicate through the VIOS to the client's network. Any tips for me. Thanks.
_files/photo.png){kind=small}
Hi burlay@i-teco.ru, Is your query resolved by now? For installing VIOS, the LPAR need to be created with lpar environment as 'vioserver'. (if creating from command line using mksyscfg command, use the option 'lpar_env=vioserver')
_files/photo.jpg){kind=small}
Hi burlay@i-teco.ru, this is could be a hardware/FSP issue. What model blade are you using? Have you logged a hardware call with IBM for this issue?
_files/photo_005.png){kind=small}
Hi Chris, I have one question in another area vios troubleshouting: 1. I installed aix on blade p-series 2. I want to change it to vios, when I try to install vios on this blade, I had error : " i/o hosting requires a hosting partition" I think partition on blade has some flag with "Aix/linux", how I chage it to "Vios"? I can't find it in ASMI and other, can you help me? burlay@i-teco.ru
_files/photo_002.png){kind=small}
Chris, I had issues configuring VLAN into the client partitions on an existing setup and I got the clue only after going through this blog where I was going wrong. The solution is bit tricky and strong to the problem of partition unable to communicate the VIOS on the same subnet. This caused us to think if there is an issue with the DLPAR capabilities and tried to play with RMC stuff, also we suspected vlan tagging on the network switches might have went wrong. Lastly, the trick was applied and all started working well. Thanks a lot for sharing this quality piece of information.
_files/photo_004.png){kind=small}
Hi chris and friend, I try to create SEA on blade system but always fail. the example flow is like this LHEA adapter ent0 LHEA adapter ent1 create etherchannel from ent0 and ent1-> ent2 create virtual adapter -> ent3 (add vlan id) create SEA adapter -> ent4 please advice... thx so much :)
One point I did not make in this post: I performed all of these tasks from the blade center console terminal session. Given that I was removing the active network interface from the VIOS, I was unable to perform the changes using an SSH session to a network interface that I needed to remove!
_files/photo_002.jpg){kind=small}
Excellent troubleshooting, Chris. I've seen VLAN tagging set up via IVM before, but given that it can only be done on the command line and the sheer complexity of it, I wonder whether they'd be better to have an HMC or SDMC anyway. Much as I love the command line, many users just want a simple solution that they can manage, or at least view, by themselves via a browser. If your only connection to the IVM is via the same adapter used for the SEA, it's pretty challenging to set up.
