Сетевые решения UCS Антон Погребняк Инструктор Fast Lane RCIS

Ports
– I/O module
 Fabric Interconnect Forwarding modes
– End-host mode (EHM)

vs Switch mode
– Dynamic and static pinning concepts
 Server Connectivity Options
– Cisco VIC1200 series
 C-Series Integration

Channel
Eth

Lossless Ethernet:
1/10GbE, FCoE, iSCSI, NAS

Use-cases

 Flexible LAN & storage

convergence based on
business needs
 Service can be adjusted based on the demand
for specific traffic

FC


Native Fibre Channel



Benefits
 Simplify switch purchase - remove
ports ratio guess work
 Increase design flexibility
 Remove specific protocol bandwidth
bottlenecks

6248: Unified Ports
Dynamic Port Allocation: Lossless Ethernet or Fibre Channel




Ports on the base card or the Unified Port GEM Module can either be Ethernet
or FC
 Only a continuous set of ports can be configured as Ethernet or FC
 Ethernet Ports have to be the 1st set of ports
 Port type changes take effect after next reboot of switch for Base board ports
or power-off/on of the GEM for GEM unified ports.

is required for the new port personality to take into

affect
Recommendation is to configure GEM card, therefore GEM is only needed to
be rebooted

8 Links
160 Gbps per Chassis
2x 2 Link
40 Gbps per Chassis
2x

1 Link
20 Gbps per Chassis

2208XP
only

for internal managment
– 10G-KR sever facing links (HIF)
– Fabric links

(NIF)
 The servers’ mezz cards use those IO channels for external connectivity
 Each IOM provides a separate dedicated IO channel for internal management
connectivity

Server





UCS-IOM-2204XP
•
Bandwidth increase for improved response esp
for bursty Applications
o
o
40G to the

Network

160G to the Host Redundant

o

(2x10G/ Half width slot; 4x10G/ Full
width slot)

•
•

Latency Lowered to 0.5us within IOM

Investment Protection with Backward and
Forward Compatibility

Server





UCS-IOM-2208XP
•
Bandwidth increase for improved response esp
for bursty Applications
o
o
80G to the

Network

320G to the Host Redundant

o

(4x10G/ Half width slot; 8x10G/ Full
width slot)

•
•

Latency Lowered to 0.5us within IOM

Investment Protection with Backward and
Forward Compatibility

up to FI
220x-XP Architecture

Fabric Ports to FI
2208
2204
WoodsideASIC








2204
Feature
2204-XP
2208-XP
ASIC


Fabric Ports
(NIF)

Host Ports
(HIF)


CoS
Woodside


4


16


8
Woodside


8


32


8
Latency
~ 500ns
~

500ns

access up

none 10G(D) --

Eth1/1/2 1 eth access down Administratively down 10G(D) --
Eth1/1/3 1 eth access up none 10G(D) --
Eth1/1/4 1 eth access down Administratively down 10G(D) --
Eth1/1/5 1 eth vntag down Link not connected 10G(D) 1365
Eth1/1/6 1 eth access down Administratively down 10G(D) --
Eth1/1/7 1 eth vntag down Link not connected 10G(D) 1369
Eth1/1/8 1 eth access down Administratively down 10G(D) --
Eth1/1/9 1 eth access down Administratively down 10G(D) --
Eth1/1/10 1 eth access down Administratively down 10G(D) --
Eth1/1/11 1 eth access down Administratively down 10G(D) --
Eth1/1/12 1 eth access down Administratively down 10G(D) –

Blade Northbound Ports


 These interfaces (show int brief – NXOS shell) are backplane traces
 Eth x/y/z where
– x = chassis number
– y = is always 1
– z = host interface port number
--------------------------------------------------------------------------------

SFP+
UCS Internal Block Diagram

UCS 6248
Expansion Module
UCS 6248
2208XP
2208XP
Double the Fabric Uplinks
Mezzanine




Server

Blade

Midplane

IO Modules

Fabric

Interconnects

Quadruple the Downlinks

2


Slot 3


Slot 4


Slot 5


Slot 6


Slot 7


Slot 8

2


Slot 3


Slot 4


Slot 5


Slot 6


Slot 7


Slot 8

you loose 1 link?
–
–
–
–
Servers’ vNIC on that link will lose

a data path.
The remaining 3 links will still pass traffic for the other blade servers
To recover the failed servers’ vNIC, re-acknowledged of the chassis is required
Since we only support 1, 2, and 4 link topologies the UCS will fall back to 2 links with
regards to blade to fabric port mapping.

on
mezzanine port mapped
to IOM 1 for blades 1
and 5

6
slot 7
slot 8
4 links, Discrète - Today



Fabric
E

Interconnect
X
 Available bandwidth per
blade – 10Gb

 Statically pinned to
individual fabric links

 Deterministic Path

8 links, Discrète


slot 1
slot 2
slot 3
slot 4
slot 5
slot 6
slot 7
slot 8
 Available bandwidth per
blade – 20Gb

 Statically pinned to
individual fabric links

 Deterministic Path

 Guaranteed 10Gb to
each blade

Up to 8 links, Port-channel

F
Fabric
Interconnect
X
 Available bandwidth per
blade – up to 160Gb

 Statically pinned to Port-
channel

 Increased and shared
bandwidth

 Higher Availability

count for NIF ports




VIC1200
adaptor with
DCE links in
Port-Channel





Gen-1 adaptor
with single 10G
link

ports (on FI) are controlled by the same Unified Port

Controller (UPC)
 Connect fabric links from IOM to the FI to the same UPC
 For fabric port-channeling, Virtual Interface (VIF) namespace varies, depending on number and
how the fabric links are connected to the FI ports.
– Connecting to the same UPC (a set of eight ports), Cisco UCS Manager maximizes the number of
VIFs used in service profiles deployed on the servers.
– If uplink connections are distributed across UPC, the VIF count is decreased. For example, if you
connect seven (IOM) fabric links to (FI) ports 1-7, but the eighth fabric link to FI port 9, the number
of available VIFs is based on 1 link – IOM port 8 to FI port 9.

Rapid
Provisioning

State abstraction




Location
Independence

Blade or Rack
Logical
Physical
vHBA
1
vNIC
1

between FC and Eth interfaces
 They also identify the origin

server
 VIFs are instantiated on the FI and correspond to frame-level tags assigned to
blade mezz cards
 A 6-byte tag (VN-Tag) is preprended by Palo and Menlo as traffic leaves the
server to identify the interface

–

VN-Tag associates frames to a VIF

 VIFs are ‘spawned off’ the server’s EthX/Y/Z interfaces (examples follow)

links  concept of virtual
interfaces (vifs) to split Eth and

FC
 Two types of VIFs: veth and vfc
– Veth for Ethernet and FCoE; vfc for FC traffic
 Each EthX/Y/Z or Po interface typically has multiple vifs attached to it to carry
traffic to and from a server
 To find all vifs associated with a EthX/Y/Z or Po interface, do this:

32 Gbps
HW Capable of 256 PCIe devices
•
OS restriction apply
PCIe virtualizationOS

independent(same as M81KR)
Single OS driver image for both M81KR and 1280 VIC
FabricFailover supported
Eth hash inputs : Source MAC Address, DestinationMAC Address,
Source Pprt, DestinationPort,Source IP address, Destination,P
address and VLAN
FC Hash inputs: Source MAC Address

DestinationMAC Address, FC SID and FC DID

UCS Cisco 1280 VIC Adapter

Customerbenefits


Dual 4x 10 GE (80 Gb per host)

VM-FEX scale, up to 112 VM interfaces /w ESX 5.0

Featuredetails
•
Dual 4x 10 GE port-channels to a single server slot

UCS 2208 IOM

Side B

Side A
UCS 1280 VIC




256 PCIe devices

UCS 2208 IOM

10 Gb FTP traffic
Connectivity IOM to Adapter


Up to

32 Gbps throughput per vNIC using flow based port-channel hash

2208 IOM

 Implicit Port-channel between UCS 1280 VIC
adapter and UCS 2208 IOM


 7-Tuple Flow based hash


 A vNIC is active on side A or B.


 A vNIC have access to up to 32 Gbps

throughput .

SFP+
16x SFP+
UCS 6248
Expansion Module
UCS 6248
2208XP







1280 VIC

x16 Gen 2

x16 Gen 2
IOH

CPU
CPU
2208XP







4x10 Gbps Ether channel from VIC 1280 to 2208 IO
Modules
 No user configuration required
 vNIC flows are 7-tuple Load Balanced across links
 Each individual flow limited to 10Gb
 Fabric Failover available

UCS Blade Chassis

Fabric

Interconnects





IO Modules



Midplane



Mezzanine




Server Blade

Default mode
–
–
–
–
–
No spanning-tree protocol (STP); no blocked ports
Admin differentiates between

server and network ports
Using dynamic (or static) server to uplink pinning
No MAC address learning except on the server ports; no unknown unicast flooding
Fabric failover (FF) for Ethernet vNICs (not available in switch mode)

 Switch mode: User configurable
– Fabric Interconnects behave like regular ethernet switches
– STP parameters are lock

of hosts
to the network
 No STP – simplifies upstream
connectivity

 All

uplinks ports are forwarding
– never blocked

VNIC 0


Server 2

L2
Switching


VNIC 0


Server 1

MAC
Learning

MAC
Learning

VLAN 10

End Host Mode



Spanning
Tree

LAN

FI A

vEth 3
Fabric A

vEth 1

Policies to prevent packet
looping
– déjà vu check
– RPF
– No uplink

to uplink forwarding

 No unknown unicast or multicast
– igmp-snooping can be disable on
per-VLAN basis

VLAN 10

Uplink Ports

FI

RPF

Deja-Vu

vEth 1






VNIC 0


Server 2

vEth 3






VNIC 0


Server 1

End Host Mode
Unicast Forwarding



LAN

Server 2

on exactly one uplink
port (or port-channel) i.e., it is
dropped when

received on other
uplinks

 Server to server multicast traffic
is locally switched

 RPF and déjà vu check also
applies for multicast traffic

Uplink
Ports


FI

vEth 3






VNIC 0


Server 1

LAN

Broadcast
Listener
per VLAN

B

vEth 1




B
VNIC 0


Server 2

B

prevent loops
– STP parameters are not
configurable
 Server vNIC traffic follows

STP
forwarding states
– Use VPC to get around blocked
ports

 VTP is not supported

 MAC address learning on both
uplinks and server links

LAN








MAC
Learning

vEth 3









VNIC 0


Server 2

vEth 1
VLAN 10


L2
Switching


VNIC 0


Server 1

Switch Mode


Root

to the uplink

 UCSM will periodically vEth
distribution and redistribute the
vEths

across the uplinks

VNIC 0


Server 1

VNIC 0


Server 2

FI A

LAN

vEth 2

vEth 1

VNIC 0


Server 3

vEth 3

Pinning

Switching

VLAN 10

failed uplinks
FI-A
VLAN 10


L2
Switching
Sub-second re-pinning
vEth 1





VNIC stays up

VNIC 0
MAC A
MAC C
Switching





vSwitch

/ N1K
ESX HOST 1
VM 1
VM 2

MAC B

vEth 3
Fabric A

All uplinks forwarding for all VLANs
GARP aided upstream convergence
No STP
Sub-second re-pinning
No server NIC disruption

3
Fabric A

More Bandwidth per Uplink
Per flow uplink diversity
No Server NIC

disruption
Fewer GARPs needed
Faster bi-directional convergence
Fewer moving parts


RECOMMENDED

VLAN 10


L2
Switching






VNIC 0

vEth 1





NIC stays up

VNIC 0
MAC A



Server 2

No disruption


No GARPs
needed

Sub-second convergence

Switching





vSwitch / N1K
ESX HOST 1
VM 1 VM 2
MAC B MAC C

pinning
 Deterministic traffic flow
 Pinning configuration is done under the
LAN

tab -> LAN Pin groups and
assigned under the vNIC
 No re-pinning with in the same FI
 Static and dynamic pinning can co-
exist

FI A

LAN

vEth 2









VNIC 0


Server 2

Pinning

vEth 1
Switching







VNIC 0


Server 1

vEth 3
VLAN 10







VNIC 0


Server 3

vEth Interfaces
vEth 1
vEth 2
vEth 3

Uplink
Blue
Blue
Purple

Administrator Pinning Definition

when
defining a service profile

 Traditionally done using NIC
bonding driver in

the OS

 Provides failover for both

unicast and multicast traffic

 Works for any OS on
bare metal and hypervisors

LAN

SAN B

SAN A

Fabric Extender





Adapter



CiMC


Half Width Blade

Fabric Extender





Adapter



CiMC


Half Width Blade

UCS Fabric
Interconnects

Chassis

Interconnect B

(no B Series
required)
Nexus 2232




PCIe Adapter

CPU

Mem
OS or Hypervisor

Nexus 2232



2 LOM ports
exclusive CIMC
connectivity

C200M2, C210M2,
C220M3, C240M3,

Adapter support:
Emulex CNA
Qlogic CNA
Intel 10g NIC
Broadcom 10g NIC
Cisco VIC

Mgmt Traffic
Data Traffic

C-Series UCSM Integration

& C Series is
supported (no B Series
required)
Nexus 2232
Nexus 2232
Mgmt Traffic
Data

Traffic

C-Series UCSM Integration
Single Wire Management with VIC1225