A data center logically includes hardware and software. The hardware refers to the infrastructure of the data center, including supporting systems and computer equipment, etc.; the software refers to the programs installed in the data center and the information services provided. A complete data center within its building includes these three logical parts: supporting systems, computer equipment, and information services. The supporting system mainly consists of power equipment, environmental adjustment equipment, and monitoring equipment. These systems are necessary conditions to ensure the normal and secure operation of the upper-level computer equipment. The computer equipment in the data center includes servers, storage devices, and network devices. These devices run the upper-level information services. The quality of information services depends on the server capabilities of the underlying support system and computer equipment. Only by comprehensively considering various factors and overall planning can the good and stable operation of the data center be ensured.
Servers, as the main carriers of information services in the data center, are connected to storage devices and network devices, making them a core component of the data center. Currently, data center servers can be divided into three categories based on their form: tower servers, rack servers, and blade servers. From a network design perspective, the deployment mode of tower servers is similar to that of rack servers. However, due to restrictions such as space in the data center server room, rack servers and blade servers have become the primary server forms in data centers. The following analysis focuses on these two types of servers.
Server Form Differences
Tower Servers
The appearance of a tower server resembles that of a personal PC host. Compared to a regular PC, the motherboard of a tower server has stronger expandability with more interfaces and slots than a regular PC, and the case size is larger than a regular PC. Tower servers have low costs and can meet the needs of entry-level servers. However, tower servers occupy a large amount of rack space and are not easy to move, so this type of server is rarely deployed in larger-scale data centers.
Figure 1. Tower Server
Rack Servers
A rack server is a server designed according to a unified standard appearance that works with cabinets, as shown in Figure 2. Due to the adoption of a uniform rack structure, the server can conveniently connect to an Ethernet switch within the same cabinet or located in the row head cabinet, simplifying the cabling and management of the server room.
The dimensions of rack servers have a unified standard: the width of the server is 19 inches, and the height is measured in U units, with the height of rack servers ranging from 1U to 7U.
Compared to tower servers, the advantages of rack servers include smaller cabinet space occupancy, more servers per unit space, easier unified management within the server room, and convenient server mobility. Rack servers require higher cooling standards for the server room. For air-cooled data centers, when installing servers, cold air is supplied through perforated floors in front of the server cabinets, and the server draws in cold air from the front panel. After passing through the internal components of the server, the cold air turns into hot air and exits from the back panel of the server. The hot air then re-enters the cooling device via a circulation channel.
Figure 2. Rack Server
Due to the limited internal space of the server, its expandability is restricted; for example, a 1U server typically has only 1 to 2 PCI expansion slots. Therefore, this type of server is mostly used in large data centers with many servers and appropriate cooling facilities.
Blade Servers
Blade servers have integrated and high-density characteristics. Its main structure is a large chassis with standard dimensions (also called "blade box"), inside which multiple "server blades" can be inserted. Each server blade is an independent server. Each server blade can run its own operating system on a local hard drive, with no interconnection between them. Multiple server blades can also form a server cluster through clustering software. In cluster mode, all server blades are interconnected through a high-speed network environment to serve the same user group. Besides server blades, different functional "blade modules" can be installed in the blade box as needed, such as network blades, storage blades, and management blades. The function of the network blade (or blade switch) is equivalent to an Ethernet switch. The blade switch usually connects to the blade server with 1GE or 10GE ports and provides 10GE ports to connect with upstream switches. The storage blade can be regarded as a hard disk module, providing storage functions to the server blade through the backplane bus or hard disk interface line. The management blade manages and monitors the blade server centrally through the monitoring management chip integrated on the server blade.
Figure 3. Blade Server
Blade servers are generally applied in large data centers or computing-intensive fields, such as telecommunications, finance, and internet data centers. For enterprises and internet service providers, as business develops and demand for servers increases, blade servers have significant advantages over rack servers in terms of saving space, ease of management, and scalability. However, due to the significantly increased computing density, blade servers impose higher requirements on the power supply capacity and cooling methods of single cabinets.
Data Center Server Network Access Overview
As shown in Figure 4, in a typical data center server area layered network model, the network core layer is used to connect the aggregation layer devices of each server area, achieving high-speed forwarding of packets between server areas. The aggregation layer devices act as gateways for servers, serving as the convergence point of network traffic in the server area, making it the best position to deploy security devices and application optimization devices. The data center access layer provides highly available Layer 2 network access for servers and achieves isolation of accessed servers through VLAN division. The number of ports on access switches should be planned according to the number of servers, considering future expansion capabilities. Access layers should be uniformly planned according to the physical form of servers. Servers of the same form have basically the same requirements for network access methods and cabinet wiring, so placing servers of the same form together and connecting them to the same access layer switch can improve the utilization rate of server cabinets and the efficiency of server room management.
Regardless of whether the server is a rack server or a blade server, high availability and high scalability are basic principles of network design at the data center network access layer. However, there are different requirements for network manageability between rack servers and blade servers. In the blade box of a blade server, there is usually a management module (or management blade). Through the WEB interface provided by the management module, unified management of the server blades and blade switches in the blade box can be achieved. Although this management unity simplifies the management of the entire system, it leads to the problem of blurred boundaries between network management and server management. Since data center network administrators do not have management permissions for blade servers, they cannot manage the blade switches; while server administrators, due to limitations in network technical capabilities, cannot effectively complete the configuration work related to the Layer 2 network access of servers, which easily causes network failures in server access. During the implementation process of a certain bank's data center project, I encountered a similar issue: under the then-existing conditions, to ensure the Layer 2 scalability and high availability of servers, the spanning tree protocol was enabled between the blade switch and the data center aggregation switch. However, due to incorrect spanning tree protocol parameters configured by the server administrator on the blade switch, the network was interrupted, affecting the go-live time of the entire business system. Therefore, clearly defining the boundary between network and server management and improving network manageability is also an important design principle when planning the network access for blade servers.
Server access layer switches can be divided into two major solutions based on whether they use N:1 virtualization technology (such as H3C IRF technology). Starting from the high availability, scalability, and ease of management of the data center network, different server forms (rack, blade) have different recommended networking solutions. The following analyzes the applicability of the two major solutions for rack servers and blade servers.
Server Access Solutions Without Virtualization Technology
There are four types of network topology schemes for virtualization technology: inverted-U shape scheme, U-shape scheme, rectangular scheme, and triangular scheme. To facilitate the comparison of various networking topologies, here we assume that the access switch (or blade switch in the blade box) uses two 10 Gigabit Ethernet interfaces to connect with the upstream Ethernet switch, and the server accesses two access switches (or blade switches) separately through dual NICs. Table 1 shows the applicability analysis of each solution.
Two-layer Loop-free U-shaped Networking
Figure 5. Two-layer Loop-free U-shaped Networking
Advantages:
There are no two-layer loops in the network access layer, and the access switch does not need to enable the Spanning Tree Protocol, so the network configuration management is simple.
Disadvantages:
There is a lack of two-layer redundant paths from the access switch to the aggregation switch, so the plan lacks high availability;
The VLANs for server access cannot cross the aggregation layer, so servers cannot achieve two-layer interconnection across switches, and the two-layer expansion capability of the network is limited.
The server gateway points to the VIP address of VRRP on the aggregation switch, but the transmission path of VRRP heartbeat packets must pass through the two access switches. When the link between the two access layer switches breaks, both aggregation switches become the master device of VRRP, causing the network to enter an unstable state at the third layer.
Applicability Analysis for Rack Servers:
The network access lacks high availability and has limited two-layer expansion capability, so it is not recommended to adopt this networking when accessing rack servers.
Applicability Analysis for Blade Servers:
The network access lacks high availability and has limited two-layer expansion capability, so it is not recommended to adopt this networking when accessing blade servers.
Two-layer Loop-free Inverted-U Shaped Networking
Figure 6. Two-layer Loop-free Inverted-U Shaped Networking
Advantages:
There are no two-layer loops in the network access layer, and the access switch does not enable the Spanning Tree Protocol, so the network configuration management is simple.
The VLANs for server access can cross aggregation switches, so they can realize VLANs across different access layer switches, allowing servers to achieve two-layer interconnection across access switches, resulting in good server access scalability.
The uplink aggregation switch of the access switch uses bundled links, so the reliability of the uplink link is high, and the bandwidth utilization rate of the link is high.
Disadvantages:
When the link between the aggregation switch and the access switch breaks, the server cannot perceive this fault, and the server's uplink traffic still sends to the faulty access switch, thus forming a "traffic black hole."
Applicability Analysis for Rack Servers:
Due to the existence of the "traffic black hole" issue, it is not recommended to adopt this networking when accessing rack servers.
Applicability Analysis for Blade Servers:
The blade switch can solve the "traffic black hole" problem through the status monitoring mechanism of the uplink bundled link: during normal operation, the blade switch periodically checks the status of the uplink aggregation layer switch interface. When an uplink interface failure is detected, the blade switch will shut down all its ports. At this point, the servers connected to this blade switch will switch their traffic to the NIC connected to another blade switch, thereby avoiding the "traffic black hole."
This plan has simple configuration management. If the blade switch has the feature of preventing "traffic black holes," it is applicable to the network access of blade switches.
Two-layer Loop Rectangular Networking
Figure 7. Two-layer Loop Rectangular Networking
Advantages:
The VLANs for server access can cross aggregation switches, so they can realize VLANs across different access switches, allowing servers to achieve two-layer interconnection across access switches, resulting in good server access scalability.
There are redundant links between the access switch and the aggregation switch, so the network access layer has high availability.
Disadvantages:
Under normal circumstances, the link between the two access switches is blocked by the Spanning Tree Protocol. When the uplink link of one access switch fails, the link between the switches becomes a forwarding state. At this moment, all the uplink traffic from the failed side of the switch will pass through the other side of the switch to the aggregation switch, increasing the uplink convergence ratio of the switch by a factor of two, leading to network congestion and reduced network forwarding performance.
Applicability Analysis for Rack Servers:
Server access has high availability and high scalability. When one side of the access switch fails, the other side of the switch becomes congested, reducing the network forwarding performance. Therefore, it is not recommended to adopt this networking when accessing rack servers.
Applicability Analysis for Blade Servers:
The blade switch module needs to configure the Spanning Tree Protocol, which is not conducive to the management and maintenance of the blade system. Similarly, there exists the problem of reduced network forwarding performance when one side of the blade switch fails, so it is not recommended to adopt this networking when accessing blade servers.
Two-layer Loop Triangular Networking
Figure 8. Two-layer Loop Triangular Networking
Advantages:
The VLANs for server access can cross aggregation switches, so they can realize VLANs across different access switches, allowing servers to achieve two-layer interconnection across access switches, resulting in good server access scalability.
There are redundant links between the access switch and the aggregation switch, so the access network has high availability, and uplink traffic sharing can be achieved through MSTP.
Disadvantages:
Network configuration management is relatively complex. To enhance the high availability and security of the two-layer network, features such as "BPDU protection," "loop protection," and "root protection" need to be enabled on the access switch and aggregation switch.
Applicability Analysis for Rack Servers:
The server access network has high availability and high scalability, so it is recommended to adopt this networking when accessing rack servers.
Applicability Analysis for Blade Servers:
The configuration on the blade switch is complex, and the manageability is poor, so it is not recommended to adopt this networking when accessing blade servers.
Analysis of Blade Server Pass-through Networking Method
Blade servers can integrate blade switches or deploy pass-through modules to extend the server's network access to the network outside the blade box. As shown in Figure 10, the focus of network access layer design for blade servers using pass-through modules is the same as for rack servers, emphasizing ensuring high availability and scalability of server access. Therefore, it is recommended to adopt the "two-layer loop triangular networking" with high availability and scalability.
Figure 9. Blade Server Pass-through Networking
Server Access Solution Using Virtualization Technology
For the access layer, traditional architectures often use MSTP+VRRP to ensure network high availability, requiring the operation of MSTP protocols between the access switch and aggregation switch, which makes configuration and management complex. When both the access switch and aggregation switch use network virtualization technology, each pair of access switches and aggregation switches can be combined into two virtual logical switches, with bundled links connecting between the logical access switch and logical aggregation switch, as shown in Figure 10.
Figure 10. Access Method Using Virtualization Technology
Advantages:
High Availability Network. Virtualization technology achieves high availability design for server access networks through N:1 backup of member devices and cross-device link aggregation.
High Scalability Network. Server access VLANs can span aggregation switches without two-layer loop issues, realizing a large two-layer server access network,