DDC Technology White Paper

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Contents

Overview·· 1

Background· 1

Weaknesses of classic switches· 1

DDC technology· 1

Technical highlights· 2

DDC implementation· 3

DDC physical architecture· 3

DDC physical devices· 3

DDC requirements for device connections· 4

DDC data forwarding mechanism·· 5

Introduction· 5

Technical highlights· 6

Generation and synchronization of forwarding entries· 7

Cell-based traffic forwarding· 9

VOQ-based congestion control 11

Procedure of cell-based forwarding· 12

Technical highlights achieved by H3C·· 13

Decentralization· 13

Openness· 14

Applications of DDC in AI data centers (AIDC) 15

 

Overview

Background

Weaknesses of classic switches

With the rapid development of big data, cloud computing, and artificial intelligence (AI) technologies, data centers need to accommodate traffic surge, which poses immense pressure on their core switches.

Traditionally, modular switches act as core switches in data centers. Each modular switch is an enclosed, centralized, and large chassis. All switch components, such as routing engines, switching fabric modules, and interface modules are housed within a large-size chassis. Although this design allows for centralized management, it has the following disadvantages:

·     Limited scalability

The port density of a classic modular switch depends on the number of chassis slots. Once the slots are exhausted, you must replace the entire switch or purchase a new one.

·     Unsatisfying energy efficiency

As switching chips advance and switching capacity increases (from 100 G to 400 G), classic modular switches have a significant increase in power consumption. For example, a 16-slot modular switch with all 400-G ports requires a power supply of up to 40000 to 50000 watts. This requirement makes upgrading devices in old data centers a significant challenge, especially when the power supply cannot meet this requirement.

 

DDC technology

Diversified Dynamic-Connectivity (DDC) is an innovative network architecture that breaks away from the classic switch design (centralized and modular switch). It adopts a distributed, decoupled approach to enhance flexibility and scalability in data center networks. This technology provides breakthroughs both in physical appearance and data forwarding.

·     Physical appearance: DDC decomposes a large modular switch into multiple fixed-port switches.

As shown in Figure 1, DDC splits a classic modular switch into smaller, independent components, fixed-port switches, which allows for distributed deployment of network functions. This process is similar as decomposing a “big iron lump” into “building blocks”, enabling distributed deployment of network functions. Those fixed-port switches act as forwarding interface modules or switching fabric modules and are deployed across multiple cabinets. This approach not only provides better cooling management and power management consumption control, but also allows flexible and convenient network deployment by eliminating limitations in device upgrade and space expansion.

Figure 1 Classic switch vs. DDC device

 

·     Data forwarding: DDC integrates the physical connections among multiple fixed-port switches into a cell-based forwarding network. In such a network, data packets are forwarded as quickly and efficiently as they are forwarded within a modular switch.

Technical highlights

DDC provides the following benefits:

·     Flexible scalability

Fixed-port switches can be freely combined as if they are building blocks. With DDC, you can add new fixed-port switches without replacing core hardware. This practice allows for dynamic network expansion to meet growing business demands.

·     Ultraspeed traffic forwarding

DDC uses advanced hardware technologies for traffic forwarding, such as Virtual Output Queuing (VOQ) and cell switching. The VOQ technology prevents packet loss during packet forwarding within the DDC system. The cell forwarding technology ensures better load sharing within the DDC system, and increases the utilization and throughput of links between fixed-port switches inside the DDC system. The two technologies works together to ensure low forwarding delay and packet loss, which meets the transport network requirements of High Performance Computing (HPC) services.

·     Extreme reliability

DDC supports subsecond-level fault recovery, perfectly fitting the requirements of large-scale AI clusters with tens of thousands of cards.

·     Green and environmental protection

The DDC architecture improves energy efficiency, because it reduces power and cooling requirements by providing support for precise capacity planning and dynamic power management.

In summary, DDC delivers exceptional flexibility, scalability, and robustness, making it an ideal solution for data center network setup.

DDC implementation

DDC physical architecture

DDC physical devices

Figure 2 DDC physical architecture

 

As shown in Figure 2, H3C's DDC solution decouples the classic design of modular switch into two types of physical devices:

·     Network connectivity fabric (NCF)

NCF functions similarly as the switching fabric module of a modular switch, which is used for transparent packet transmission. If the input and output interfaces of a packet are located on different NCPs, the related NCF forwards that packet from the ingress NCP to the egress NCP.

NCFs do not have service interfaces and use SerDes Framer Interfaces (SFI) to transmit packets between NCPs.

·     Network connectivity processor (NCP)

NCP functions similarly as the service module and MPU of a modular switch, which is responsible for processing protocol packets and forwarding service packets.

NCPs use SFIs for NCF connection.

The service ports of an NCP are connected to service networks, allowing the DDC system to communicate with external networks.

Table 1 shows the port matrix for the S12500AI switch series.

Table 1 Port matrix for the S12500AI switch series

Product series	Chassis type	Product model	Available interfaces
H3C S12500AI switch series	NCF	S12500AI-128EP-NCFN	·     128 x OSFP800 SFIs
	NCP	S12500AI-36DH20EP-NCPN	·     20 x OSFP800 SFIs ·     36 x QSFP 112 service interfaces
		S12500AI-18EP20EP-NCPN	·     20 x OSFP800 SFIs ·     18 x OSFP800 service interfaces

 

NOTE: As a best practice to enhance network reliability, make sure the DDC system contains a minimum of two physical devices for each DDC role (including NCP and NCF).

 

DDC requirements for device connections

In a DDC system, messages between NCPs are transparently transmitted through NCFs, with control and data packets sharing transmission channels. The connection requirements are as follows:

·     Each NCP is connected directly to the NCFs via SFIs. No additional connections are required between NCPs or between NCFs.

·     Each NCF has 128 SFIs, and each NCP has 20 SFIs. An NCF can provide interconnectivity for up to six NCPs.

·     As a best practice to enhance network reliability, make sure the DDC system contains a minimum of two physical devices for each DDC role (NCC, NCP and NCF). Each NCP should be connected to a minimum of two NCFs.

Figure 3 DDC network topology

 

DDC data forwarding mechanism

NOTE: In the current software version, DDC supports only Layer 3 forwarding. This document describes only the forwarding mechanism of DDC.

 

Introduction

By decoupling a modular switch into multiple fixed-port switches, DDC inherits fixed-port devices’ advantages, including flexible scalability and easy maintenance and upgrade. Meanwhile, DDC also distributes the pressure of weight load, power supply, and cooling across the fixed-port devices. However, the core technical challenge lies in: how to the packet forwarding efficiency of modular switches when distributed fixed-port switches are used to forward service traffic? This is addressed by DDC's innovative data forwarding mechanism. 

As shown in Figure 4, the classic Spine-Leaf architecture builds a standard Ethernet forwarding network with service interfaces. Based on SFIs and internal protocol interactions, the interconnected NCPs and NCFs in a DDC system automatically form a cell-based forwarding network. In this network, traffic is forwarded through cell-based forwarding. For devices outside the DDC system, those NCPs and NCFs function as a single device at the forwarding plane.

·     Limitations of the classic Spine-Leaf architecture

¡     Relies on 5-tuple to implement per-flow ECMP load balancing, which is prone to uneven traffic distribution (caused by hash collisions), especially in scenarios with a large amount of traffic (for example, elephant flows). This practice might cause congestion on some links while other links remain idle.

¡     To avoid traffic congestion, you must reserve bandwidth and adjust the oversubscription ratio, which lowers resource utilization.

·     DDC's innovative solution

¡     Cell switching: Divides packets into fixed-size cells and forwards them in parallel through multiple paths, achieving efficient load balancing across nodes (NCPs).

¡     Intelligent VOQ scheduling: NCPs dynamically perceive path bandwidth before forwarding and perform load balancing only over available paths. This practice ensures non-blocking switching and maximizes link utilization.

Figure 4 Data forwarding in classic Spine-Leaf architecture vs. Data forwarding in DDC

 

Technical highlights

Cell switching is used inside a DDC system, eliminating hash polarization for all traffic patterns. 100% non-blocking transmission is achieved within the cell network.

In ALL-to-ALL scenarios, DDC significantly improves data transmission performance, with the average forwarding efficiency 2.5% higher than counterpart solutions.

 

NOTE: ·     In data centers, ALL-to-ALL (full mesh) communication refers to a mode where all compute nodes (for example, GPU/CPU servers) in a cluster must exchange data frequently with every other node. This is one of the core challenges in high-performance scenarios, including AI training and distributed computing. ·     InfiniBand (IB) is a interconnect technology featuring high performance and low latency, which is primarily used in data center, supercomputer, and high-performance computing (HPC) environments. Through dedicated hardware and protocol stacks, this technology delivers much higher bandwidth and efficiency than traditional networks (Ethernet), making it ideal for large-scale networks that require concurrent communication.

 

Generation and synchronization of forwarding entries

Cell forwarding table generation on NCFs

In a cell network, NCFs forward data as guided by the cell forwarding table. To achieve unified management of the entire cell network, DDC requires the network administrator to configure a unique member ID (slot number) for each device during network deployment. For NCF and NCP forwarding chips, this ID is uniformly referred to as MODID.

Both NCFs and NCPs use dedicated forwarding chips. Only SFIs support cell-based traffic forwarding are SFI interfaces. After an NCP is connected to an NCF, the NCF automatically discovers the NCP, acquires its MODID, records the NCP-facing port, and then generates a local cell forwarding entry. This entry contains the mapping between MODID and output interface, as shown in Table 2.

Table 2 Cell forwarding entries on an NCF

MOD (slot number)	Output interface
MOD-ID 1	Int1
MOD-ID 2	Int2
MOD-ID 3	Int3
MOD-ID 4	Int4

 

Cell forwarding table generation and synchronization on NCPs

To achieve unified management of the cell network, DDC assigns a globally unique Systemport identifier to each NCP service interface. The Systemport ID is in MODXPortY format, which uniquely identifies a service interface within the cell network. The MODX portion represents the member ID and the PortY portion represents the port number. For example, MODXPortY represents the Y-th service interface on the NCP with a member ID of X.

In a cell network, the following forwarding actions are supported:

·     Local forwarding: When the input and output interfaces for a traffic flow reside on the same NCP, local forwarding is performed in the same way as traditional IP forwarding.

·     Cross-device forwarding: When traffic needs to be forwarded across NCPs, each NCP uses the globally synchronized forwarding table (as shown in Table 3) to complete packet forwarding.

Table 3 Cell forwarding entries on an NCP

Prefix	Interface/SystemPort	Remote encapsulation index
IP1/32	Int1	N/A
IP2/32	MOD2Port1	EncapIndexX
IP3/32	MOD3Port1	EncapIndexY
IP4/32	MOD4Port1	EncapIndexZ

 

You might question: How does an NCP generate cell forwarding entries? In fact, each NCP is pre-configured at factory with dedicated CPU virtual interfaces (OSF interfaces). All OSF interfaces are interconnected via a Layer 2 network, forming a unified broadcast domain. DDC requires all OSF interfaces to be placed on the same IP subnet, and builds a BGP EVPN network accordingly. With BGP peer relationships established in that BGP EVPN network, DDC achieves device interconnects at the control plane.

As shown in Figure 5, NCPs generate and synchronize forwarding entries as follows:

2.     Phase 1: Forwarding entry generation

When NCP 1 connects to Server 1, NCP 1 automatically generates the following entries after ARP learning and route discovery:

¡     ARP entry: Records the mapping between IP1 and its MAC address.

¡     Route entry: Marks the access port as MOD1Port1 and assigns a remote encapsulation index of EncapIndexX. The remote encapsulation index corresponds to certain encapsulation information required for Layer 3 forwarding, such as destination MAC address and VLAN information.

3.     Phase 2: Forwarding entry synchronization

¡     NCP 1 synchronizes 3-tuple (IP1, MOD1Port1, EncapIndexX) information with the other NCPs through BGP EVPN routes.

¡     Similarly, when NCP 4 connects to Server 2, it synchronizes 3-tuple (IP4, MOD4Port1, EncapIndexY) information with the others.

¡     Upon receiving 3-tuple information, each NCP generates local ARP and FIB entries accordingly, ultimately achieving synchronization of forwarding entries across all NCPs.

4.     Phase 3: Network-wide consistency in forwarding entries

Finally, all NCPs achieve consistency in data forwarding entries, ensuring accurate cross-NCP traffic forwarding.

This forwarding mechanism ensures that all NCPs in the cell network can correctly implement both local and cross-device forwarding, which allows for efficient NCP collaboration.

Figure 5 Cell forwarding table generation on NCPs

 

Key highlights of the DDC control plane

·     Unique identifier (Systemport): Prevents port conflicts and simplifies management.

·     EVPN route synchronization: Achieves distributed forwarding table synchronization through standard BGP, supporting large-scale expansion.

·     Remote encapsulation index (EncapIndex): Decouples forwarding and control planes, improving hardware forwarding efficiency.

Cell-based traffic forwarding

Application scenarios

DDC uses the cell-based forwarding technology to accommodate scenarios where massive data processing and high-speed data transmission are required, such as cloud data centers, HPC, AI training platforms, and large-scale machine learning.

Technical highlights

The cell-based forwarding technology has the following advantages:

·     Optimized link utilization: Cell-based forwarding reduces or avoids traffic congestion caused by uneven distribution of traffic. Unlike traditional ECMP, which might unevenly distribute a large amount of traffic across a few links, cell-based forwarding ensures that traffic is evenly distributed across all available links, regardless of the traffic size. This forwarding method improves overall network performance by reducing uneven link utilization (for example, scenarios where some links are overloaded and others remain idle).

·     Less network congestion: Traffic load is distributed more evenly across links, which lowers the risk of network congestion. This is crucial for maintaining low end-to-end latency and high throughput.

·     Higher DC resilience and robustness: Cell-based forwarding provides better traffic control, enabling data centers to effectively process massive traffic and traffic bursts. This enhances network stability and reliability.

Operating mechanism

Cell-based forwarding involves the following steps:

1.     Traffic division

When a data flow arrives at the ingress of a data center, it is divided into smaller units, which are then encapsulated as cells. As basic units of data transmission, cells can be managed and routed independently from the original data flow. Cell forwarding information includes SystemPort, remote encapsulation index, and cell serial number (SN).

2.     Independent routing

Each cell is hashed independently based on its header information. After hashing, cells are distributed across multiple equal-cost paths based on the hash result. Unlike traditional flow-based ECMP load balancing, cell-based load balancing optimizes link utilization by allowing for finer control of data flow distribution.

3.     Cell reassembly

Before data reaches the target node, the related cells are reassembled into the original data flow at the destination for final processing or storage.

Figure 6 Data forwarding within a DDC system

 

VOQ-based congestion control

Application scenarios

VOQ-based congestion control is widely used in large-scale data centers, high performance computing, and big data processing, especially scenarios that require high network performance and stability.

Technical highlights

·     Smooth data transmission: With precise traffic scheduling, data is transmitted smoothly within a DDC system.

·     Priority-based traffic scheduling: VOQ supports priority-based traffic scheduling, ensuring that prioritized data flows are transmitted first.

·     Optimized bandwidth utilization: VOQ improves the overall bandwidth utilization by avoiding traffic congestion and optimizing data transmission paths.

Operating mechanism

VOQ is a high-performance queue management technology for congestion control. Multiple virtual output queues correspond to multiple egress ports. After bandwidth resource acquisition, virtual output queues are flexibly scheduled to complete traffic forwarding.

Figure 7 VOQ operating mechanism

 

As shown in Figure 7, VOQ operates as follows:

2.     Credit application and assignment

After an NCP starts, it exchanges messages with the connected NCFs to synchronize DDC topology information, creates virtual output queues according to the number of DDC egress ports, and then generates VOQ entries as shown in Table 4.

Table 4 VOQ entries on NCPs

Systemport	Output interface
MOD2Port1	VOQ1
MOD3Port1	VOQ2
MOD4Port1	VOQ3

 

After the ingress NCP receives packets, it categorizes those packets, looks up the related route, and then places those packets into the virtual output queue corresponding to the target egress port and packet priority. Instead of delivering those packets to the related NCF and egress NCP, the ingress NCP cooperates with the egress NCP through VOQ to identify whether bandwidth resources are sufficient.

¡     The virtual output queues at the ingress port request credit resources from the egress port to notify that some data needs to be forwarded out of the egress port.

¡     The egress NCP assigns credit resources to the ingress port only if the egress port has sufficient bandwidth resources.

¡     If the egress port does not have sufficient resources, it will not allocate credit resources to the ingress port. Without obtaining a credit, a virtual output queue temporarily stores packets locally, and sends them only after it obtains bandwidth resources. This avoids packet congestion and loss within the DDC system.

3.     Data forwarding

After the ingress port receives a credit, it forwards packets to the target NCF based on the related VOQ entry. This process involves dividing the packets into cells and distributing those cells across all available paths.

Procedure of cell-based forwarding

Data is forwarded on the DDC data plane as follows: (In this example, packets are forwarded from Server1 to Server4):

1.     When Server1 sends a packet to Server4, NCP1 first looks up the related cell forwarding entry to obtain the Systemport and remote encapsulation index.

2.     Based on the SystemPort, NCP1 adds the packet to the related VOQ. If available credits exist, the packet is segmented and encapsulated as cells, and then sent to the upstream NCF through the related SFI interface.

3.     Upon receiving those cells, the NCF looks up the related local cell forwarding entry based on the cell forwarding information carried in those cells, determines the output interface, and then forwards the cells to NCP4.

4.     After NCP4 receives the cells, it reassembles the packet based on the received cell forwarding information, obtains Layer 2 packet header encapsulation information, encapsulates the reassembled packet, and then sends it to Server4 through the designated port.

Figure 8 Data forwarding in a cell network

 

Technical highlights achieved by H3C

Decentralization

Typically, non-H3C DDC solutions use a centralized architecture, where a controller node (NCC) centrally manages and controls the system. The NCC works in conjunction with service modules (NCP), switching fabric modules (NCF), and optional management switches (MGT) to form a three-tier architecture. H3C, through groundbreaking innovation, adopts a design that integrates control and forwarding, to streamline the three-tier architecture into a two-tier architecture (NCP + NCF). This decentralized solution fully retains DDC functions while significantly enhancing system reliability and scalability. It is a better partner for the evolution of cloud-native and distributed networks.

Table 5 H3C DDC solution

Item	H3C DDC solution
Component	·     NCP (Service module) ·     NCP (Switching fabric module)
Forwarding performance	Same performance (TTL-1), comparable to classic switches
Network architecture	·     Decentralized architecture ·     No single point of failure ·     Simplified deployment with lower costs
Application scenarios	Scenarios where minimal networking is required, for example, Cloud/distributed networks and SDN

 

Figure 9 H3C DDC solution

 

Openness

In building an open ecosystem, the DDC technology has established an open interoperability framework based on standard BGP. BGP extensions are made to advertise Tunnel Egress Point (TEP) information, which formulates unified communication standards across vendors.

H3C and industry partners jointly define the core DDC framework standards based on the Open Scheduling Framework (OSF) for AI networks, providing end-to-end standardized guidance from requirements analysis and architecture design to technical implementation. This can accelerates the deployment of the technology ecosystem.

The DDC standardization design delivers the following technical benefits:

·     Vendor unlocked: Standardized interfaces enable seamless collaboration among hardware from different vendors, realizing "free hardware definition.”

·     Open and win-win: This solution drives the industry from closed to collaborative, lowering the barrier for ecosystem participation.

Figure 10 Participation of H3C in DDC standardization

 

 

Applications of DDC in AI data centers (AIDC)

AI data center (AIDC) leverages heterogeneous parallel computing technologies, as well as GPU/TPU clusters and high-performance networks, to implement large-scale AI training, inference, and data analysis. Thanks to its powerful computing support and efficient resource scheduling, AIDC is becoming the core architecture of modern data centers.

In AIDC scenarios, DDC adopts an innovative network architecture that overcomes limitations of classic modular switches, enhancing flexibility and scalability of AI computing networks through distributed decoupling. This design breaks large network devices into independent modules (for example, NCP and NCF), enabling decentralized deployment of network functions while optimizing cooling and power supply management. It delivers high bandwidth and low latency to satisfy AI tasks.

Furthermore, DDC significantly improves network performance and reliability in AIDC, offering new approaches for scaling up AI computing clusters. Integrated with H3C’s intelligent management and control system, this solution enables efficient resource orchestration and automated operations, greatly reducing management complexity.

DDC also employs a series of high-performance technologies, such as non-blocking forwarding architecture, VOQ, and cell switching, delivering far superior forwarding efficiency and bandwidth utilization compared to classic solutions. These technologies balance AI training traffic (for example, AllReduce communication), prevent network congestion caused by data synchronization between GPUs, and maximize overall performance of distributed AI tasks.

Figure 11 Applications of DDC in AIDC network