Cisco Nexus Fex Windows Server 2016 Lacp
Hello,I am planning some new gear deployment and I wanted to verify that the provisioning process would work as expected (unfortunatley don't have an extra pair of switches to test it now).So situation is fairly simple - bunch of 1U servers that will be uplinked to pair of Cisco 9Ks via LACP bond. Add a little twist - I also need to PXE boot them. I know of 'no lacp suspend-individual' command that would bring up a port even if there's no LACP received on it.What I don't know and can't find with the help of Google - how will this command behave in vPC environment?
Can it be different on 2 sides of vPC? If yes, I can just use it on 1st switch - if port is not sending LACP then the port will come up on it, while port on 2nd switch will still be disabled.
If not - are there any other tricks? Or am I SOL and will have to write some additional automation to change state of ports after the servers are built?Any information will be greatly appreciated!. I just setup a pair of 93180s a few weeks back and ran into a similar situation. We configured the port channel with LACP connected through a vPC. That command successfully solved the issue I had where the ports would go into suspended status almost immediately as they came up. By different do you mean that the LACP settings are different on each side of the vPC? I'm not clear.I set the no lacp suspend-individual on the port channel to both switches.
Cisco Nexus Fex Configuration
If I didn't have it on the other switch the port on that switch would go into suspended because it isn't receiving LACP PDUs. That command successfully solved the issue I had where the ports would go into suspended status almost immediately as they came upI have a different use case - I need to PXE boot a server, which has no config.
So I MUST have only 1 port active to avoid any loops or other storms (not so much during PXE boot, but in case somebody makes accidental config change later).Also, I am not sure why you would ever need to do it. If you configured LACP bonding properly on your server (on Linux it's bond mode 4, I believe) - it comes up just fine with LACP.
If you used bond mode 1, just configure port-channel as static aggregate (mode on instead of mode active/passive). I presume that you already read this related to PXE booting off of LACP bonded links via vPC? Dying light save game corrupted fix.
Windows Server 2016 Lacp
It was for the 7Ks, but I found it still applicable to troubleshooting the same error of not receiving PDUs on the 9Ks.The vPC ought to take care of any potential loops. I'm running a pair of N9Ks right now with the interface on each switch of the vPC in independent mode and not getting spanning tree giving me fits. On an older Nexus on NXOS 5.x I didn't need to explicitly add the command, but when I jumped to a newer switch running NXOS 7.x the problem became that 9K it didn't want to work anymore without explictly disabling it from suspending the interfaces. I tried putting the Nexus side of the port channel into on mode, but the problem is that the other was expecting to negotiate so while the interface wouldn't go into suspended anymore we couldn't pass traffic to the box.
I presume that you already read this old thread related to PXE booting off of LACP bonded links via vPC?Yes, thanks, but also I have found this:'As per the warning guys, this could cause you HUGE problems if you enable this on a port that is part of a vPC, so I would only use this no lacp suspend-individual on ports that are not part of a vPC port channel'Hence I am wondering if anybody tried this config.when I jumped to a newer switch running NXOS 7.x the problem became that 9K it didn't want to work anymoreAh, interesting, I didn't work with 7.x yet, thanks for heads up. Did you try LACP fast/slow settings?. I'm not sure I grok the concern here.During PXE, you'll have two links up, but only one of them will have an IP stack up and running. NIC 1 can't make any use of NIC 2, let alone bridging a frame from one side to the other.After the OS is up, the worst case scenario is that your OS will treat the NICs as individuals, both will be up, and it'll bridge traffic from one to the other. But STP (particularly with edge port configuration) should mitigate that.Couple of anecdotes:I've used one switching OS that allows the configuration on the individual interfaces to differ from the configuration on the channel.
So, totally different interface configuration would apply with vs. Without LACP.If you load iPXE onto the ROMs of your server NIC, the whole problem goes away because there's a slow-protocols (LACP) module in iPXE.edit: The warning message in the article you cited is: Nexus(config-if)# no lacp suspend-individualWarning:!! Disable lacp suspend-individual only on port-channel with edge ports. Disabling this on network port port-channel could lead to loops.!You're talking about an edge port. No problem?.
The worst case scenario is that your OS will treat the NICs as individuals, both will be up, and it'll bridge traffic from one to the otherIt is also possible that the host will run virtualization and will have some kind of virtual switching, and I am not sure how that would work.Ultimately, I don't want to go through every possible scenario of what could get wrong to prove that having 2 ports in port-channel up is OK. I know that plugging in Cisco UCS FI (that falls back to individual if no LACP is present) to static port-channel results in spectacular network failures and that knowledge is enough for me to state that I am going to ensure that if LACP is not working, I won't have more than 1 port of port-channel active. No.The port runs as Individual. It acts as though the channel-group command wasn't configured on the port.Yes, I'm sure.NX-OS is nice that it's configurable in this regard.
The options are:. Only operate if we're sending/receiving LACP PDUs without error / refuse to run without them. Operate as a normal switchport in the absence of LACP PDUs. / Operate as a LAG member when LACPs are flowing correctly.Other platforms (Cat4K) do not have the ability to operate Independently.Yet other platforms (Cat2K) do not have the ability to refuse to operate without LACP PDUs. They'll happily operate as Individual ports without them.
Symptom:pair of B22 Fexes dual-homed to a pair of Nexus 6000 running 7.0(6)N1(1).Server connected by eVPC with LACP NIC teaming, running Windows HyperVWhen the server disables the NIC team member, the fex reports the link is still up. When the server re-enables the port, connectivity is broken.Shut/no shut on the Fex side or reconfiguring the Fex port or power-cycling the Fex does not help.Reloading the server resolves the issue.Keeping the same HW configuration, if the same server is booted with Linux OS, there is no issue: the Fex detects the link down.Replacing the Fex with a pass-through module, when the Windows server disables one NIC, the parent switch correctly detects the link down.Conditions:Server running Hyper V NIC teaming and running Windows.