1. Background
Previously, we detailed Bilibili's exploration of its customized data center (R2-AZ2) project [1], focusing primarily on the technological iteration and implementation of smart energy-efficient data centers. The efficient operation of a data center does not exist in isolation; it relies on a complex and sophisticated interconnection network to ensure connectivity between servers, storage, and network equipment within the data center.
The cabling system is a key component for realizing data center interconnection. Poor management of data center cabling can lead to prolonged delivery cycles for production environments, excessive reserved cables, chaotic cable layout, difficult equipment installation, increased troubleshooting and maintenance time, and even affect the airflow organization of cabinets, causing local overheating and compromising the safe operation of electronic information equipment.
Additionally, with the rapid development of AI technology and business applications, intelligent computing centers are rising rapidly, and networks are evolving toward higher bandwidth, lower latency, and lower power consumption. This means that requirements for network and cabling systems are continuously increasing.
As one of the key infrastructures of large-scale data centers, how to use digital management tools to improve the efficiency of delivery, operation and maintenance of cabling systems has also been a question we have been considering and a direction for our exploration and practice.
1.1 Introduction to Data Center Cabling
Before conducting structured cabling design for a data center production system, it is necessary to first understand the business side's network requirements, determine the network cabling topology, and finalize the data center structured cabling design plan based on the designed network topology [2].
Figure 1 Network Cabling Implementation Flowchart
Two common cabling methods are used: structured cabling and unstructured cabling [3].
Structured cabling uses standardized predefined connection points and path designs, leveraging one or more wiring areas and cabling products (such as cabling network cabinets, high-density patch panels, etc.) to provide standards-based connections for electronic information equipment. When electronic information equipment is moved, added, or changed, only the cables need to be re-patched in the centralized wiring area, while the permanent link portion remains unchanged regardless of equipment movement, addition, or modification. Structured cabling systems use high-density pre-terminated optical fiber cables, greatly reducing the number of optical fibers in the machine room, but the design and installation of the cabling system require more time and cost, and the addition of intermediate connection points in the same link may reduce network transmission quality.
Unstructured cabling, also known as point-to-point cabling, as the name suggests, this type of cabling system does not use any predefined standards, connection points, or paths, and directly performs end-to-end cable connections. Compared with structured cabling, unstructured cabling systems have lower installation costs and shorter installation times, but have higher later operation and modification costs.
Bilibili currently flexibly chooses cabling methods based on different machine room environments and business network requirements, comprehensively considering factors such as cost, delivery cycle, and on-site environment. For conventional server cabinet areas, unstructured cabling is usually prioritized, while for high-priority network core cabinet areas, structured cabling design will be considered to optimize cable quantity and later operation and maintenance management.
Figure 2 On-site cabling diagram for network core cabinets
1.2 Development of Bilibili Cabling Intelligent Management Platform
For data center network cabling tasks in a single building, they are usually completed in multiple batches of cabling projects according to the business delivery rhythm. When handling each batch of cabling projects, full-life-cycle information flow and management rely on personnel of different roles manually completing tasks through office software tools. Facing the rapid delivery demands of today's complex medium and large-scale data center cluster production systems, there are the following problems and challenges:
(1) File standardization and accuracy issues: During the cabling project, insufficient standardization and accuracy of document flow may lead to information loss or errors. We need to ensure that file formats, data content, etc., meet certain standards and reduce errors caused by manual operations.
(2) Route planning efficiency issues: The current manual planning method has low efficiency and is prone to incorrect route length estimation, leading to errors and extra workload in actual construction. We need to improve the automation and accuracy of route planning to reduce manual intervention and errors.
(3) High cabling management complexity: Due to the large number of personnel, roles, and basic data involved, cabling management has become a tedious and long-cycle task. Traditional single-batch cabling projects and delivery information become isolated, leading to incomplete historical cabinet cabling status and information, which is not conducive to improving the delivery efficiency of the machine room production environment and the operation and maintenance management and cabling optimization after commissioning. We need to establish more efficient information management and data sharing mechanisms to achieve smoother information transmission and more optimized cabling strategies.
Based on the above background and combined with our actual cabling project management experience, we propose the development project of Bilibili's data center network cabling intelligent management platform, aiming to digitize, visualize, and intelligentize the offline network cabling process, enabling professional technicians to complete cabling planning, management, and operation and maintenance work easily, quickly, and accurately.
2. Cabling Intelligent Management Platform
For the Bilibili cabling management platform, we have sorted out the complete cabling management process, as shown in the following flow diagram.
The Bilibili cabling intelligent function platform conducts rapid development and iteration by function modules and in phases. This article will focus on the cabling route automation module marked in the red box, as the accuracy of this part of the data will directly affect the final effect of the entire cabling task.
2.1 Function 1: Basic Information Maintenance
The objects involved in cabling management include but are not limited to the following information:
Cables
Cabinets
Side cabinets
Aisles
Columns
Row head cabinets
Row tail cabinets
Room-specific cable trays
Cross-room production cable trays
……
For the above basic data, we extract the data strongly related to cabling projects, integrate and abstract several data models, such as cabinet model, aisle model, cross-room interconnection model, etc. Combined with the cabinet floor plan & production cable tray floor plan, operation and maintenance personnel can maintain the above basic data information in advance according to different physical environment conditions, laying a foundation for cabling project calculations.
2.2 Function 2: Automated Cabling Route Calculation
Method 1: Scenario Induction Calculation Method
To automatically complete cabling route calculation, Bilibili uses a cost-effective method to quickly build a simple and clear schematic diagram of the data center room layout. Operation and maintenance personnel can draw the physical topology structure of the room by adding rows and columns, associate basic data, and establish the relative positional relationships of columns, aisles, and cabinets.
Figure 5 Display of the machine room layout interface built on the platform
Summarizing daily cabling requirements, we have integrated and abstracted 6 high-frequency cabling scenarios, which are:
Cabling in the same room and same aisle
Cross-aisle cabling in the same room [specified route direction]/[unspecified route direction]
Cross-room cabling without specified route, where the start cabinet and end cabinet are in the same row as the room cable tray outlet
Cross-room cabling without specified route, where either the start cabinet or the end cabinet is not in the same row as the room cable tray outlet
Cross-room cabling with specified route, where neither the start cabinet nor the end cabinet is in the same row as the room cable tray outlet
Other scenarios can be added flexibly.
For different scenarios, define a unified formula for calculating cabling distance, and select the shortest cabling route.
Method 2: Shortest Path Optimization Method
Cabling route calculation can be transformed into solving the optimal path problem in graph theory. Compared with the scenario induction calculation method of Method 1, it has higher calculation efficiency and is suitable for a wider range of machine room layouts and cabling scenarios, but the difficulty lies in how to transform actual engineering problems into mathematical problems for solving.
We first number power and low-voltage cable trays, machine room cabinets, and power distribution cabinets in sections and in an orderly manner during the design phase. Then, during the cabling planning phase, we establish mathematical relationship models for equipment involved in cable connections such as cable trays, cabinets, and power distribution cabinets, clarify their geometric relationships with each other, and input initial data on the cabling length of each section of the cable tray. Finally, through the optimal path planning algorithm, we automatically calculate the shortest cabling path between any equipment and output the planned path results.
Figure 6 Shortest Path Optimization Flowchart
2.3 Function 3: New Cabling Task
2.3.1 Task Generation
Generate a single cabling requirement on the platform in the form of a task. For a single cabling task, you can batch create cabling details according to cabling scenarios (such as vertical cabling in cabinets, cross-cabinet cabling from business switches to network cores, cross-cabinet cabling between network cores, etc.).
The following core information is clearly specified in the cabling details:
Cabling task name: Naming to distinguish task requirements
Cabling start point information: including start room, start cabinet, start U position
Cabling end point information: including end room, end cabinet, end U position
Cable type: Category 6 unshielded twisted pair (UTP) cable, LC-LC, AOC, MPO……
Cable number: Automatically generated
……
After confirming the basic cabling information, the system will automatically calculate the shortest path and output the cabling length, providing a detailed list of all cabling tasks.
Figure 7 Display of the new cabling detail interface on the platform
2.3.2 Task Execution
After the platform cabling task is generated, enter the task execution phase. Through workflow approval, digitize and platformize process management documents, and update cabling project-related information in real time until the final cabling acceptance and delivery.
2.4 Function 4: Cabling Operation and Maintenance Management
After the previous cabling tasks are completed, the basic cabling information data will continue to provide value for subsequent cabling operation and maintenance management. Through visualization tools, you can quickly understand the key cabling information of the machine room and judge the delivery status of network cabling in the business environment. Through data analysis, you can further analyze indicators such as whether cable types match connected device ports, failure rates of different types/brands/batches of cables, and cabling cost trends, helping to locate faulty lines.
Figure 8 Schematic diagram of cabling operation and maintenance management visualization interface
3. Platform Application and Future Outlook
Currently, version 1.0 of Bilibili's cabling intelligent management platform has been officially launched. Our submitted invention patent "Machine Room Cabling Method and System" is under substantive examination. Functions such as platformization of cabling and machine room information, automated calculation of cabling routes and lengths have been realized. This not only greatly improves the efficiency and effect of cabling project implementation, but also the route length automatically planned by the platform is reduced by an average of 10% compared with the previous manual calculation length, helping network and data center operation and maintenance managers reduce cabling costs and manage cabling systems.
In the future, we will continue to integrate the experience of cabling project implementation, delivery, and network operation and maintenance, and strive to connect and apply information between network management, IDC operation and maintenance management, and cabling management. Through this approach, we will build a more comprehensive, intelligent, and powerful Bilibili cabling management platform, helping to achieve rapid delivery, intelligent operation and maintenance, and refined management of network cabling.
References:
[1] Practical Practice of Bilibili's Customized New-generation Smart Energy-saving Data Center Project
[2] Lu Lei. Research on Data Center Network Structured Cabling Technology. Telecommunications Express: Networks and Communications 8(2019):5.
[3] Which Cabling Method Should Data Centers Use? (https://mp.weixin.qq.com/s?__biz=MzA4NDkwNDI2OQ==&mid=2651020930&idx=1&sn=fcfbd25aaf24026365e3e53546a5524c&scene=21#wechat_redirect)
This is a discussion topic separated from the original topic at https://www.bilibili.com/read/cv34203492/