Cisco MDS: Long-Distance ISL Buffer Allocation

The command used to check the current buffers allocated for the port-group on the switch is:

#show port-resources module x

(where x is the module number)

Long-Distance ISLs
When long-distance ISLs are required, you must have sufficient BB_credits configured to ensure that the ISL can run at maximum capacity.
The simplest formula or the guideline for computing BB_credits for long-distance ISL assumes a full sized Fibre Channel frame of approximately 2 KB and factors in the interface operating speed and one-way distance of the ISL.

As per the table, to operate a 16-Gbps Fibre Channel ISL over 50 km, you would multiply the one-way distance (50) times the minimum number of BB_credits per km (8).
That is, a 50 km 16-Gbps ISL requires 400 BB_credits when the average frame size is approximately 2 KB.
This is the minimum number of BB_credits that are required for the link to function at its best when used to its maximum.

To accommodate a smaller average frame size than the maximum (full sized) value, more BB_credits would be required proportionally.
Since each buffer is for a Fibre Channel frame irrespective of its size, when Fibre Channel frames are not full sized, more BB_credits are required to achieve full link utilization. In this case, an approximate yet simple formula for calculating BB_credits is the following:

BB_credits = (Minimum number of BB_credits required per km for interface speed x One-way distance (km)) / ((Average receive frame size (bytes) / 2150 bytes))

The following example displays the BB_credits calculation for a 16 Gbps link that is 50 km long with an average input frame size of approximately 1 KB (1075 bytes):
(8 BB_credits per km at 16 Gbps x 50 km) / (1075 / 2150)
400/0.5 = 800 BB_credits

To take into consideration, the actual average input frame size first determines the average frame size by dividing the total input bytes by the total frames input.
The average frame size must be determined for the input direction (Rx side) on an interface since the receive BB_credits are set.
The total bytes and frames can be viewed in the show interface counters command output.

switch# show interface fc 2/7 counters
fc2/7
    5 minutes input rate 1048060640 bits/sec, 131007580 bytes/sec, 94786 frames/sec
    5 minutes output rate 253368512 bits/sec, 31671064 bytes/sec, 47717 frames/sec
    14079632456 frames input, 18624775031572 bytes
      0 discards, 0 errors, 0 CRC/FCS
      0 unknown class, 0 too long, 0 too short
    8089598629 frames output, 6040401816628 bytes
      0 discards, 0 errors
    0 timeout discards, 0 credit loss
    0 input OLS, 0 LRR, 0 NOS, 0 loop inits
    0 output OLS, 0 LRR, 0 NOS, 0 loop inits
    0 link failures, 0 sync losses, 0 signal losses
    15031 Transmit B2B credit transitions to zero
    0 Receive B2B credit transitions to zero
    11192 2.5us TxWait due to lack of transmit credits
    Percentage TxWait not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
    500 receive B2B credit remaining
    481 transmit B2B credit remaining
    481 low priority transmit B2B credit remaining
    Last clearing of "show interface" counters: 2d09h

In the above example, the calculation for the average frame size is:
18624775031572 (bytes) / 14079632456 (frames) = 1323 bytes/frame which is approximately 1.3 KB/frame

To complete the calculation:
(8 BB_credits per km at 16 Gbps x 50 km) / (1323 average bytes per frame / 2150 bytes)
400/0.61= 650 BB_credits

Thus, for a 16 Gbps link traversing 50 km carrying an average frame size of 1323 bytes would take a minimum of 650 BB_credits.

Extended buffer-to-buffer credits are made possible by allocating extra buffers to specific interfaces. These extra buffers are taken from the unallocated buffer pool.

The ENTERPRISE_PKG license is required to use extended buffer-to-buffer credits on 16 Gbps, 32 Gbps, and 64 Gbps switching modules.

To allow most buffers to be available, perform these steps:

• Configure ISL ports over different port-groups and modules.
• Configure ports that are connected to end devices (F ports) from mode auto to mode F by using the switchport mode f command.

To allocate maximum BB_credits to port fc1/5, perform these steps:

• Configure ports fc1/6-8 in the port-group to a minimum BB_Credit of 1:

  switch# configure
  Enter configuration commands, one per line.  End with CNTL/Z.
  switch(config)# interface fc1/6-8
  switch(config-if)# switchport fcrxbbcredit 1

• Configure port fc1/5 with the maximum BB_credits of 253:

  switch# configure
  Enter configuration commands, one per line.  End with CNTL/Z.
  switch(config)# interface fc1/5
  switch(config-if)# switchport fcrxbbcredit 253

• Verify the BB_credits allocation on port fc1/5:

  switch# show port-resources module 1
  .
  Port-Group 2
  Available dedicated buffers are 0
  --------------------------------------------------------------------
  Interfaces in the Port-Group    B2B Credit  Bandwidth  Rate Mode
                                    Buffers     (Gbps)
  --------------------------------------------------------------------
  fc1/5                               253       16.0     dedicated
  fc1/6                                 1       16.0      dedicated
  fc1/7                                 1       16.0      dedicated
  fc1/8                                 1       16.0      dedicated

Reference Link: https://www.cisco.com/c/en/us/td/docs/dcn/mds9000/sw/9x/configuration/interfaces/cisco-mds-9000-nx-os-interfaces-configuration-guide-9x/configuring_interface_buffers.html