Using the link aggregation feature in DDOS can result in an increase in throughput, however depending on several factors it could result in the opposite. The purpose of this article is to help determine and implement the appropriate configuration of Link Aggregation. In DD OS, since dynamic port participation methods like PAGp and MLT aren't supported, it's crucial to statically configure LAG ports on the switch.
Determine Appropriate Link Aggregation
The following is what should be considered when trying to decide on aggregation. If aggregation isn’t required, consider failover. Therefore, the last choice given is failover as an alternative to aggregation.
Direct Connect:
1. mode round-robin (if supported by the media servers)
2. separate NIC per media server (if there are enough NICs)
3. mode xor-L3L4
4. failover (if aggregation cannot be used)
Private Network:
1. separate NIC per media server (if there are enough NICs)
2. mode xor-L3L4
3. mode xor-L2 (when the client count is suitable)
4. failover (if aggregation cannot be used)
Local Network:
1. Separate NIC per media server (if there are enough NICs)
2. Mode xor-L3L4
3. Mode xor-L2 (when the client count is suitable)
4. Failover (if aggregation cannot be used)
Remote Network:
1. Separate NIC per media server (if there are enough NICs)
2. Mode xor-L3L4
3. Failover (if aggregation cannot be used)
Implementation of Data Domain Link Aggregation and Failover
Link aggregation methods supported by Data Domain:
Round Robin and
Balanced-xor (setup manually on both sides)
LACP(starting in 5.0 for 1 Gb and 5.1 for 10Gb)
The balanced-xor aggregation is selected by choosing the specific hash that is supported:
Layer 2 or
Layer 3+4
Layer 2+3(starting in 5.0)
There are four virtual interfaces that can be used to define the aggregation or failover:
veth0
veth1
veth2
veth3
Before aggregation can be used, a virtual interface must be created. This can be done by running the following command:
net create virtual <veth#>
Example:
net create virtual veth0
After a virtual interface is created any of the physical links that are available on the system can be included in the aggregation group: eth0, eth1, eth2, eth3, so forth. The onboard links (eht0 and eth1) have only recently been allowed to be added. Older installations of the Data Domain software may not allow those two links to be aggregated.
To specify aggregation of eth2 and eth3 in the virtual interface veth0, one of the following commands would be used:
net aggregate add veth0 mode round-robin interfaces eth2 eth3
The first network packet sent to veth0 will be forwarded to one of the interfaces, and the next packet would be forwarded to the other. Sending of packets continues to alternate between the interfaces until there are no more packets, or a link fails. If eth3 loses physical connection, all packets are sent through eth2 until the eth3 link is brought back up. To make this effective, the other side of the network must also be set up to do round robin. For direct connect (the only topology that is recommended for round robin) the media server must be able to be set up (and support) round robin.
net aggregate add veth0 mode xor-L2 interfaces eth2 eth3
The aggregation used would be balanced-xor. The packets are distributed across eth2 and eth3 based on XOR of the source and destination MAC addresses. Because there are only two links to be aggregated the lowest bit is used to determine the interface to use for the packet. If the result is 0, one interface is chosen. If the result is 1, the other interface is used. To get the packets to be spread across the two links requires that the data is sent to more than one destination, and the MAC addresses of the destination must be different in such a way that XOR results provide a different number. This means that one address must be odd and the other must be even. If there are three links that are aggregate, the XOR result is split three ways. There has to be at least two media servers there must be at least two media servers with odd and even MAC addresses to get any aggregation at all. In general, this aggregation should not be used with less than four media servers.
net aggregate add veth0 mode xor-L3L4 interfaces eth2 eth3
The aggregation used with this command will also be balanced-xor. Eth2 and eth3 distribute packets using XOR of source and destination IP addresses and port numbers. The result gives a number in which the lowest bit is used to determine which link to use to send the packet. An even result goes over one, and an odd result goes over the other. With three links, the result is divided by 3 with the remainder determining which interface to use. This aggregation would be used when there are a lot of connections (there is one connection per stream) or a lot of media servers or both. This is the mode of choice for Data Domain, but some switches do not support this type of hashing.
net aggregate add veth0 mode LACP hash xor-L3L4 interfaces eth4a eth4b
The aggregation used with this command will also be lacp-xor. Eth2 and eth3 distribute packets using XOR of source and destination IP addresses and port numbers. The data flow control follows the same mechanism used by balanced mode except it adds a control protocol to monitor the interfaces with a minimal amount of automated administration of the interfaces including better sensing when an interface fails. The sensing goes beyond the sensing of carrier loss to the sensing of the ability to send and receive data. The heartbeat can be sent out every second or every 30 seconds. The default is every 30 seconds. The interval determines how fast the bonding senses the link is no longer communicating and will stop using the interface. Once every 30 seconds is less invasive, but it takes longer to mark the link as down, and there may be connection timeouts while it is waiting.
net failover add veth0 interfaces eth2 eth3
This is not aggregation, but the command will group together interfaces eth2 and eth3 for failover. There is only one failover type supported. If the active physical link goes away the data is sent to the second physical link. The active interface is determined by which link comes up first when it is set up. This is nondeterministic. It depends on several factors such as switch activity, network activity, and which interface is brought up first when they are enabled. The active interface can be determined by specifying one of the links as primary. The primary interface will always be set as active if it is UP and RUNNING.
Note: For DDBOOST/OST backups, use ifgroup instead of aggregation for better throughput and performance.
Configure and enable the virtual interface.
After the virtual interface has been created (and the right aggregation method assigned), the interface must be configured with an IP address (and subnet mask), and then enabled. The following command can be used to do accomplish this:
#net config <veth#> <IP address> netmask <subnet address> up
Example:
#net config veth1 192.168.192.168 netmask 255.255.255.0 up