ProductsMulti-Core Distributed NetStream Technology White Paper
Keyword: NetStream, NDE, NDA, NSC, Traffic statistics, accounting, ToS, NetFlow, multi-core, distributed
Abstract: This document introduces H3C’s multi-core distributed NetStream technology. NetStream is a technology dealing with network stream statistics and report. It collects statistics on network communication volume and resource usage status, and delivers these statistics data for traffic management, analysis, and accounting use. H3C’s NetStream supports multi-core CPU and distributed processing, largely improving the network stream analyzing and processing capability of the whole device, and therefore can provide users with an economical and large-capability NetStream solution.
Acronyms:
Acronym | Full spelling |
— | — |
Table of Contents
1.3 Technology Introduction. 4
2 Technical Features of Multi-Core Distributed NetStream.. 5
2.1 Fully Distributed NetStream.. 5
2.3 Independent Statistics on Inbound and Outbound Interfaces. 8
2.4.2 Aging Triggered by FIN or RST Packet of TCP. 9
2.4.3 Aging Triggered by Bytes Overflow. 9
2.5 NetStream Statistics Output 10
3 NetStream Management Application Tool 13
4.4 Application Monitoring and Analysis. 16
4.5 User Monitoring and Analysis. 17
1 NetStream Introduction
1.1 Overview
With the fast development of network software and hardware technologies, networks support more and more services and applications, and users get a higher network bandwidth. To perform more precise management and accounting of network traffic, users bring higher requirements on traffic analysis. H3C’s NetStream is based on network stream statistics and report. It collects statistics on network traffic and resource usage status, and delivers relevant reports, through which network administrators can know about the detailed information of the network traffic. The NetStream delivered data can be used for network management and layout, enterprise and branch accounting, ISP billing, data reservation, and other data collections for business purposes.
As shown in the following figure, NetStream works at the IP layer. In the process of IP datagram transfer, NetStream classifies and collects statistics of the network packets transferred between the IP layer and the data link layer (such as Ethernet, PPP, HDLC, and Frame Relay) on a per-stream basis, and then sends the statistics in UDP packets of a specific version to the NetStream Collector (NSC), and then to the NetStream Data Analyzer (NDA), which then analyzes the statistics and delivers the related reports.
Figure 1 Logical position of NetStream in a network device
1.2 Concepts
l NetStream: Network stream, a set of packets with the same characteristics
l NDE: NetStream Data Exporter
l NSC: NetStream Collector
l NDA: NetStream Data Analyzer
l AS: Autonomous System, which is applied in BGP
l ToS: Type of Service, a flag in the IP header, used to control traffic
1.3 Technology Introduction
NetStream defines the data packets transferred from source to destination in a period of time as a network stream. NetStream identifies the characteristic of a network stream through IP address and TCP or UDP port number; together with IP protocol type, ToS, and inbound (or outbound) interface, they uniquely identify a network stream. That is, a network stream is determined by the following:
l Source IP address
l Destination IP address
l Source port number
l Destination port number
l IP protocol type
l ToS
l Inbound/outbound interface
Figure 2 NetStream data collection and analysis
As shown in the above figure, a typical NetStream system consists of NDE, NSC, and NDA.
1) NDE
NDE is usually a router. The NDE uses a stream buffer to maintain the related information and processing rules of network streams. According to the first packet of a network stream, NDE establishes a processing rule and uses the rule to process the succeeding network streams, collects the matching stream statistics and then exports the statistics to the NDC. The NDE can aggregate the data before exporting the data to the NDC.
2) NSC
NSC is usually an application running on a Unix or Windows system, used to resolve the packets from NDE, collect the statistics, and put the statistics data into the database, so that the NDA can analyze the data. The NSC can collect the exported data of multiple NDEs, and further filter and aggregate the data.
3) NDA
NDA is a network traffic analyzer. It gets statistics data from the NDC for further processing, and generates related reports, providing reference for various services (such as traffic accounting, network planning, and attack monitoring). Usually, the NDA has a graphical user interface. Users can get, view, and analyze the data collected by the NSC.
2 Technical Features of Multi-Core Distributed NetStream
2.1 Fully Distributed NetStream
At present, the NetStream function (NDE) is implemented on a distributed device by using a loose-coupling NetStream service board. According to the configured rules, each IO board mirrors the specific traffic to the NetStream board for preprocessing (including classification and aggregation), which then sends the statistics information to NSC and NDA for further processing.
Table 1 Comparison between NetStream implementation methods
| Centralized processing on the NetStream board | H3C SR6608 NetStream multi-core distributed processing |
Performance | The throughput of a distributed device is usually very large. Mirroring a large amount of traffic to the NetStream service board for centralized processing may result in performance bottleneck of the board. | Fully distributed NetStream: NetStream is enabled on a Layer 3 interface of an interface board; traffic analysis is performed on the interface board. Therefore, performance bottleneck is avoided, and the NetStream processing ability of the whole device is improved. |
Deployment cost | Users need to purchase a special service board for NetStream. This causes a high deployment cost and occupies the precious slot resources as well. | No NetStream board is needed, saving the user cost. |
As described in the above table, the traditional centralized processing of NetStream on a special service board is available in function implementation but has defect in performance and cost; while the fully distributed multi-core NetStream solution implemented on H3C’s SR6608 products well solves the above problems.
2.2 Multi-Core NetStream
For a network device with a single CPU, packet processing and command configuration are all done by the single CPU. The device performance cannot be improved largely due to the limited CPU processing capability. The more services supported, the bigger affection on the functions at the control level (such as slow responses to configurations). Besides, it is harder and harder to improve the device performance simply by increasing the CPU main frequency for technical reasons. Although ASIC and NP techniques have high processing capability, they are bad in general-utility and complex-service processing, and need a long research and development period, therefore cannot satisfy the requirements on flexible multi-service processing and fast service providing.
Multi-core CPU is addressed to solve the above problems.
Multiple generic CPUs and some functional components integrated into a chip form an SOC (System on Chip), which can be called a multi-core CPU. Communications between the CPUs and between a CPU and a functional component on the chip are achieved through the internal high-speed interconnection technology, which breaks the performance bottleneck of the communications between the CPUs in a multi-CPU system and the communications between the CPUs and other components of the system. Multi-core CPU can process tasks concurrently and therefore greatly improves the system performance.
On PCs for home entertainment and businesses, Intel and AMD’s dual-core CPU is applied successfully. They do very well in concurrent processing of tasks. What’s more, four-core CPU with better performance has already been developed and applied.
At present, multi-core CPU is stepped onto the stage of the communication devices. H3C takes the lead and brings users the multi-core routing devices, that is, the SR6600 series routers, for business uses. The H3C SR6600 series routers adopt a multi-core multi-threaded CPU to forward network packets and process tasks concurrently at a high speed. The multi-core CPU of the SR6600 routers has eight CPU cores, each having four hardware threads. Multiple threads share the level-1 cache line, and each thread has its own hardware register. The whole system therefore will run as if it has 32 CPUs (8 × 4) processing tasks concurrently.
Figure 3 Software architecture of H3C’s multi-core NetStream
The above figure is a general view of the H3C’s multi-core NetStream software architecture.
This architecture features the following:
l Effective load balancing: The hardware packet distribution engine distributes the packets to the service CPU cores equally and at a high speed, avoiding the overloading or underloading of a service CPU core.
l Powerful concurrent network stream analysis: The multi-core software architecture based NetStream NDE can concurrently run on multiple service CPU cores to analyze and aggregate the network traffic according to certain rules, and export related statistics to the NSC for further processing. The NetStream capability of the device is improved largely thanks to the concurrent processing of tasks.
2.3 Independent Statistics on Inbound and Outbound Interfaces
NetStream collects statistics on the inbound and outbound interfaces respectively.
For an outbound interface: A network stream is identified by the outbound interface, source IP address, destination IP address, source port number, destination port number, protocol number, and ToS.
For an inbound interface: A network stream is identified by the inbound interface, source IP address, destination IP address, source port number, destination port number, protocol number, and ToS.
2.4 Statistics Aging
In real networking, a large volume of traffic may occur in a short period, while the NDE (a router) has a limited memory space, so the NDE uses a specific algorithm to delete some streams from its NetStream buffer to release the memory space. Meanwhile, the NDE outputs the statistics of these streams to the NSC in UDP datagrams. This process is called statistics aging.
H3C’s NetStream provides four aging mechanisms, aging by time, aging triggered by FIN and RST packet of TCP, aging triggered by bytes overflow, and aging by CLI. The following sections describe the four aging mechanisms in detail.
2.4.1 Aging by Time
Usually, network streams in a network may burst out. For the NDE, a hundred thousands of packets that belong to a stream may pass through the NDE in the first 60 seconds, but there may be no such packets passing through in the second 60 seconds.
To deal with this, NetStream provides two aging methods by time:
l Aging by inactive time: Age the stream if there are no such packets passing through in the specified inactive timeout.
l Aging by active time: Age the stream if the specified active timeout is reached since the first packet of the stream passed through.
H3C’s NetStream supports the two aging methods, and you can configure the aging timeout as needed.
2.4.2 Aging Triggered by FIN or RST Packet of TCP
For a TCP connection, a session is terminated if either party sends a packet with the FIN or RST flag set. Therefore, if a FIN or RST packet passes through an existing TCP stream, the NDE will immediately age the stream, and output the steam statistics to the NSC. However, if the first packet of a stream is the FIN or RST packet of TCP, the NDE will not age the stream but create the stream normally and proceed according to the normal flow.
2.4.3 Aging Triggered by Bytes Overflow
The total number of packets passed of a stream is counted in the NetStream buffer. For a typical 32-byte system, the upper limit is about 4G for integer. When the total number of bytes reaches the upper limit, if statistics collection on the number of bytes proceeds, bytes overflow and therefore statistics error will occur. To avoid bytes overflow, H3C NetStream will age a stream immediately if it finds that the stream reaches 3.9G bytes, and output the stream statistics to the NSC.
2.4.4 Aging by CLI
H3C NetStream allows you to execute related commands through the console to age all the streams in the NetStream buffer, and output the statistics information to the NSC.
2.5 NetStream Statistics Output
The NDE will send the stream statistics information in UDP datagrams to the NSC.
The NSC can check the packet loss information by supporting the sequence number.
Figure 4 NetStream packet format
At present, NetStream packets mainly have three formats, v5/v8/v9. The following table describes the advantages and disadvantages of each format.
Table 2 NetStream packet format comparison
Version | Advantages | Disadvantages |
V5 | l Abundant output fields; all fields of the stream statistics before aggregation can be output to the NSC. l Small workload for the NDE | l The packet format is fixed and inextensible. l Data volume is large. The NSC cannot hold the information for a long time, and therefore bring heavy workload of data analysis for the NSC and NDA. |
V8 | l Aggregate stream according to certain rules; data volume is relatively small. l The carried contents are simple, and suitable for specific analysis. l New aggregation modes can be added. | l The packet format is fixed and inextensible. l The NDE takes charge for the aggregation work, and therefore has a heavy workload. l If new aggregation mode is added, both the NDE and NSC need version upgrade. |
V9 | Template based l Allow to output only the needed field statistics, reducing the data volume of the output streams, and therefore reducing the possible memory and bandwidth cost of the NDE and NSC. l A new field can be added into the output statistics without changing the output packet format. Because statistics output is based on template, NSC can resolve the stream statistics even if it cannot understand the syntax of the added field. l Flexible output modes: It can output the stream statistics before and after the aggregation. |
|
& Note:
At present, H3C’s multi-core distributed NetStream supports statistics packets in the format of v5 and v8, and will support packets in the format of v9 in the near future.
1. Statistics packets in the format of v5
A v5 NetStream packet carries the original information of each stream. It includes a packet header and several packet records, each corresponding to an aged NetStream stream.
2. Statistics packets in the format of v8
If NetStream v5 format is adopted, large amount of NetStream output packets will be output when network traffic is large. To reduce the affection, NetStream v8 format classifies and aggregates the original information of the streams according to certain rules, and then sends the aggregate information. UDP version 8 NetStream statistics packets have a flag for differentiating NetStream inbound and outbound statistics.
NetStream aggregation has obvious advantages:
l It reduces the NetStream output data volume, therefore reducing the bandwidth occupied by statistics transmission between routers and network management devices.
l It reduces the workload of and the performance requirements on an NSC device.
At present, H3C multi-core distributed NetStream supports the following five types of aggregations:
Table 3 Aggregation types supported by H3C multi-core distributed NetStream
Aggregation type | Aggregation classification rule |
Autonomous system aggregation: as | Classifies network streams according to the source AS number, destination AS number, inbound interface index, and outbound interface index |
Protocol-port aggregation: protocol-port | Classifies network streams according to the protocol number, source port, and destination port |
Source prefix aggregation: source-prefix | Classifies network streams according to the source AS number, source mask length, source prefix, and inbound interface index |
Destination prefix aggregation: destination-prefix | Classifies network streams according to the destination AS number, destination mask length, destination prefix, and outbound interface index |
Source and destination prefix aggregation: prefix | Classifies network streams according to the source AS number, destination AS number, source mask length, destination mask length, source prefix, destination prefix, inbound interface index, and outbound interface index |
3. Statistics packets in the format of v9
The v5 and v8 formats lack flexibility. V9 format adopts template to solve this problem.
NetStream v9 format adopts two types of data, statistics data and option data, together with NSC/NDA to finish stream statistics.
2.6 Compatibility
H3C NetStream version 5 inbound statistics output is completely compatible with Cisco NetFlow version 5 statistics information output.
H3C NetStream version 8 inbound statistics output is completely compatible with Cisco NetFlow version 8 statistics information output.
3 NetStream Management Application Tool
H3C provides a powerful NetStream management application tool, XLog.
XLog is an extensible network analysis system. Working together with the H3C NetStream feature, it can provide the following functions:
l Constructs an extensible and distributed NetStream data stream collection and analysis system.
l Collects and stores data of multiple NDEs. Filters, aggregates the data, and stores the data into the database.
l Further analyzes the data and generates the traffic report. Adopts Web-based access and provides visual and graphical management interfaces. All the data output can be displayed on the Web pages.
XLog provides users a reliable and convenient network stream analysis solution. It can help the administrator know the running status of the internal network of an enterprise, find and solve the network performance bottleneck and abnormities in time. Besides, XLog can help users with network planning, optimizing, and troubleshooting.
For more information about XLog, refer to the related user manual.
4 NetStream Applications
NetStream technology is based on streams. You can collect detailed network information through NetStream. The NetStream function is provided by H3C series routers and switches, working in conjunction with H3C XLog to provide many applications for users.
4.1 Accounting
Figure 5 NetStream used in ISP traffic splitting
NetStream provides precise data for accounting based on resources (such as bandwidth, time range, and so on) occupation. The data includes IP address, number of packets, number of bytes, time, ToS, and application type. Internet service providers (ISP) can apply flexible accounting policies, such as accounting based on time, bandwidth, application, and service quality. Enterprise users can use these data as a reference to calculate and allocate the department costs, so as to use resources effectively.
The network illustrated in the above figure is accessed to the Internet through an H3C SR66 router and two ISPs. After NetStream is configured on the SR66, the two ISPs can calculate traffic for accounting separately according to the NetStream statistics information.
4.2 Network Planning
Figure 6 NetStream used in network planning
As shown in the above figure, NetStream can provide the key information for XLog, such as the network traffic between each AS domain, to facilitate network designing and planning, so as to implement the best network performance and reliability with the least network running cost.
4.3 Network Monitoring
Figure 7 NetStream used in network monitoring
As shown in the above figure, most of the LANs will access the Internet through a router. By deploying NetStream on the outbound interface of the router, you can monitor the traffic on the Internet outbound interface in real time, analyze the bandwidth occupation of each service, monitor restricted Internet accesses, and give a warning on the XLog in time when there are attacks, so that the administrator can analyze and troubleshoot the problems more easily.
4.4 Application Monitoring and Analysis
You can get the detailed network application information through the NetStream technology. For example, the network administrator can display the percentage of the Web, FTP, Telnet, and other commonly used TCP/IP applications to the total network traffic. Internet content and service providers can plan and allocate the network and application resources according to the information to satisfy users to the maximal.
As shown in the following figure, H3C XLog can analyze the NetStream information, collect statistics based on applications, and display the statistics in GUIs. At present, XLog supports almost 300 types of applications by default. What’s more, you can add applications according to your network status.
Figure 8 NetStream used in application monitoring and analysis
4.5 User Monitoring and Analysis
Figure 9 NetStream used in user monitoring and analysis
The administrator can easily get network and application resources usage of users in detail through NetStream, and therefore can effectively plan and allocate network resources to ensure the secure running of the network. The above figure is the specific users’ network statistics acquired by H3C XLog through NetStream.
5 Summary
Today, data network operating management is more and more important. The multi-core distributed NetStream technology can well solve the problems such as performance bottleneck and high deployment cost of the traditional traffic analysis solutions. H3C SR6608 routers that adopt this technology can help network operators know the traffic details of networks with large traffic; therefore the administrator can apply the limited bandwidth to the most valuable applications, and make the network operate in good order and with high efficiency.
Copyright ©2007-2008 Hangzhou H3C Technologies Co., Ltd. All rights reserved.
No part of this manual may be reproduced or transmitted in any form or by any means without prior written consent of Hangzhou H3C Technologies Co., Ltd.
The information in this document is subject to change without notice.