H3C S12500X-AF
NetStream Technical Topics

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Document version: 6W100-20190821

 

Contents

Overview·· 1

Technical background· 1

Benefits· 1

NetStream implementation· 1

NetStream architecture· 1

Concepts· 2

Flow aging· 2

NetStream data export 3

NetStream filtering· 5

NetStream sampling· 5

Application scenarios· 5

Example: Configuring NetStream traditional data export 5

Network configuration· 5

Configuring the device· 6

Configuring IMC· 6

Verifying the configuration· 10

Example: Configuring NetStream aggregation data export 13

Network configuration· 13

Configuring the device· 13

Configuring IMC· 14

Verifying the configuration· 14

 

Overview

Technical background

With the ever-growing numbers of services and applications provided at increasing bandwidths, traditional technologies such as SNMP and port mirroring can no longer satisfy the desired traffic accounting requirements. A new traffic accounting technology is needed to provide fine-grained traffic statistics at reasonable costs and with minimal impact on the network performance.

The NetStream accounting technology is developed to address these issues. NetStream provides highly granular per-flow traffic statistics on the device. A flow is a unidirectional set of packets that arrive at the device on the same interface, have the same source and destination IP addresses, Layer 4 protocol, TCP/UDP source and destination ports, and the same type of service (ToS) byte in the IP headers. The device accumulates NetStream statistics in the cache and can export them to an external device for further processing and analysis.

NetStream can be deployed flexibly on devices at the access, distribution, and core layers of the network to collect traffic statistics that meet specific requirements. The NetStream statistics can be used for a wide variety of purposes such as traffic monitoring and accounting, performance and network anomaly detection and troubleshooting, and network analysis and planning.

Benefits

NetStream provides the following capabilities:

·     Accounting—NetStream provides highly granular traffic statistics for ISPs to implement flexible resource usage accounting and billing based on criteria such as time ranges, bandwidth, applications, and ToS.

·     Network planning—NetStream provides valuable information, such as traffic statistics between ASs, for network management and planning. Administrators can make network capacity planning and resource allocation decisions based on the NetStream statistics to achieve optimal network performance and availability at minimum operating costs.

·     Network monitoring—NetStream deployed at the egress point of a network can collect Internet-bound traffic statistics in real time for application-based bandwidth usage analysis. The information can be used to locate inappropriate network design and network performance bottlenecks to assist administrators in network resource optimization.

·     User behavior monitoring and analysis—NetStream provides administrators with easy access to user network behavior and application usage information for securing the network and optimizing resource allocation.

NetStream implementation

NetStream provides the ability to parse the Layer 2 through Layer 4 headers, MPLS labels, and VXLAN UDP headers of packets for per-flow based traffic accounting. It aggregates packets into flows and exports the flow records to at least one NetStream data collector-typically a server that does the actual traffic analysis. Administrators can also configure NetStream to aggregate flows based on predefined criteria to meet specific monitoring and analysis requirements.

NetStream architecture

A typical NetStream system includes the following elements:

·     NetStream data exporter—A device configured with NetStream. The NDE provides the following functions:

¡     Classifies traffic flows by using the 7-tuple elements.

¡     Collects statistics from the classified flows.

¡     Aggregates and exports the statistics to the NSC.

·     NetStream collector—A program running over an operating system. The NSC parses the traffic data received from the NDEs, and saves the data to its database. The NSC can collect data from multiple NEDs.

·     NetStream data analyzer—A network traffic analyzing tool. Based on the data in NSC, the NDA generates reports for traffic billing, network planning, and attack detection and monitoring. The NDA can collect data from multiple NSCs. Typically, the NDA features a Web-based system for easy operation.

NSC and NDA are typically integrated into a NetStream server.

Figure 1 NetStream system

 

Concepts

Flow aging

NetStream uses flow aging to enable the NDE to export NetStream data to NetStream servers. NetStream creates a NetStream entry for each flow for storing the flow statistics in the cache.

When a flow is aged out, the NDE performs the following operations:

·     Exports the summarized data to NetStream servers in a NetStream export format.

·     Clears NetStream entry information in the cache.

NetStream supports both periodical aging and forced aging. The following information describes the flow aging methods in detail.

Periodical aging

Periodical aging uses the following methods:

·     Inactive flow aging—A flow is inactive if no packet arrives for the NetStream entry within the inactive flow aging timer. When the timer expires, the following events occur:

¡     The inactive flow entry is aged out.

¡     The statistics of the flow are sent to NetStream servers and are cleared in the cache.

When you use the inactive flow aging method, the cache is large enough for new flow entries.

·     Active flow aging—A flow is active if packets arrive for the NetStream entry within the active flow aging timer. When the timer expires, the statistics of the active flow are exported to NetStream servers. The device continues to collect active flow statistics.

This method periodically exports the statistics of active flows to NetStream servers.

Forced aging

To implement forced aging, use one of the following methods:

·     Clear the NetStream cache immediately. All entries in the cache are aged out and exported to NetStream servers.

·     Specify the upper limit for cached entries and configure the system to take either of the following actions when the limit is reached:

¡     Age out the oldest entries.

NetStream data export

Traditional data export

Traditional NetStream collects the statistics of each flow and exports the statistics to NetStream servers.

This method consumes more bandwidth and CPU than the aggregation method, and it requires a large cache size.

Aggregation data export

NetStream aggregation merges the flow statistics according to the aggregation criteria of an aggregation mode, and it sends the summarized data to NetStream servers. The NetStream aggregation data export uses less bandwidth than the traditional data export.

Table 1 lists the available aggregation modes. In each mode, the system merges statistics for multiple flows into statistics for one aggregate flow if each aggregation criterion is of the same value. The system records the statistics for the aggregate flow. These aggregation modes work independently and can take effect concurrently.

For example, when the aggregation mode configured on the NDE is protocol-port, NetStream aggregates the statistics of flow entries by protocol number, source port, and destination port. Four NetStream entries record four TCP flows with the same destination address, source port, and destination port, but with different source addresses. In the aggregation mode, only one NetStream aggregation entry is created and sent to NetStream servers.

Table 1 NetStream aggregation modes

Aggregation mode	Aggregation criteria
Protocol-port aggregation	·     Protocol number. ·     Source port. ·     Destination port. ·     VXLAN ID.
Source-prefix aggregation	·     Source AS number. ·     Source address mask length. ·     Source prefix (source network address). ·     Inbound interface index. ·     VXLAN ID.
Destination-prefix aggregation	·     Destination AS number. ·     Destination address mask length. ·     Destination prefix (destination network address). ·     Outbound interface index. ·     VXLAN ID.
Prefix aggregation	·     Source AS number. ·     Destination AS number. ·     Source address mask length. ·     Destination address mask length. ·     Source prefix. ·     Destination prefix. ·     Inbound interface index. ·     Outbound interface index. ·     VXLAN ID.
Prefix-port aggregation	·     Source prefix. ·     Destination prefix. ·     Source address mask length. ·     Destination address mask length. ·     ToS. ·     Protocol number. ·     Source port. ·     Destination port. ·     Inbound interface index. ·     Outbound interface index. ·     VXLAN ID.
ToS-source-prefix aggregation	·     ToS. ·     Source AS number. ·     Source prefix. ·     Source address mask length. ·     Inbound interface index. ·     VXLAN ID.
ToS-destination-prefix aggregation	·     ToS. ·     Destination AS number. ·     Destination address mask length. ·     Destination prefix. ·     Outbound interface index. ·     VXLAN ID.
ToS-prefix aggregation	·     ToS. ·     Source AS number. ·     Source prefix. ·     Source address mask length. ·     Destination AS number. ·     Destination address mask length. ·     Destination prefix. ·     Inbound interface index. ·     Outbound interface index. ·     VXLAN ID.
ToS-protocol-port aggregation	·     ToS. ·     Protocol type. ·     Source port. ·     Destination port. ·     Inbound interface index. ·     Outbound interface index. ·     VXLAN ID.

 

NetStream export formats

NetStream exports data in UDP datagrams in one of the following formats:

·     Version 5—Exports original statistics collected based on the 7-tuple elements and does not support the NetStream aggregation data export. The packet format is fixed and cannot be extended.

·     Version 9—Based on a template that can be configured according to the template formats defined in RFCs. Version 9 supports exporting the NetStream aggregation data and collecting statistics about BGP next hop and MPLS packets.

·     Version 10—Similar to version 9. The difference between version 9 and version 10 is that the version 10 export format is compliant with the IPFIX standard.

NetStream filtering

NetStream filtering uses an ACL to identify packets. Whether NetStream collects data for identified packets depends on the action in the matching rule.

·     NetStream collects data for packets that match permit rules in the ACL.

·     NetStream does not collect data for packets that match deny rules in the ACL.

NetStream sampling

NetStream sampling collects statistics on fewer packets and is useful when the network has a large amount of traffic. NetStream on sampled traffic lessens the impact on the device's performance.

Application scenarios

Example: Configuring NetStream traditional data export

Network configuration

As shown in Figure 2, configure NetStream on the device to collect and export per-flow traffic statistics as follows:

·     Collect both incoming and outgoing traffic flow statistics on FortyGigE 1/0/1.

·     Export the collected traffic statistics to the IMC server with IP address 12.110.2.2 and UDP port 5000.

The IMC platform version running on the IMC server is PLAT 7.3 (E0504).

Figure 2 Network diagram

 

 

Configuring the device

# Assign an IP address to each interface, as shown in Figure 2. (Details not shown.)

# Enable NetStream for incoming and outgoing traffic on FortyGigE 1/0/1.

<Device> system-view

[Device] interface fortygige 1/0/1

[Device-FortyGigE1/0/1] ip netstream inbound

[Device-FortyGigE1/0/1] ip netstream outbound

[Device-FortyGigE1/0/1] quit

# Specify the IMC server as the NetStream data export destination.

[Device] ip netstream export host 12.110.2.2 5000

Configuring IMC

Adding the device to IMC NTA

1.     Log in to IMC.

2.     Click the Service tab.

3.     From the left navigation tree, select Traffic Analysis and Audit > Settings.

4.     On the Settings page that opens, click Server Management.

The Device Management page opens.

5.     Click Add.

6.     On the Add Device page shown in Figure 3, perform the following tasks:

a.     Enter the device IP address (12.110.2.1) in the Device IP field, or click Select next to Device IP field to select the device and click OK.

b.     Configure other parameters as needed.

c.     Click OK.

Figure 3 Adding the device to IMC NTA

 

Deploying NTA server configuration to the device

1.     On the Settings page, click Server Management.

The Server List page opens.

2.     Click the Modify icon  for the NTA server.

3.     On the Server Configuration page shown in Figure 4, configure the following parameters:

a.     Set port 5000 as a listening port for the server.

b.     Select the device (12.110.2.1) in the Traffic Analysis area.

c.     Click Deploy.

Figure 4 Server Configuration

 

Adding a traffic analysis task

1.     On the Settings page, click Traffic Analysis Task Management.

The Traffic Analysis Task Management page opens.

2.     Click Add.

The Select Task Type page opens.

3.     Select Interface and click Next.

The Add Traffic Analysis Task page opens.

4.     In the Basic Information area, configure the following settings:

¡     Task Name—Enter a task name. This example uses Interface.

¡     Reader—Click Select next to the Reader field, select the operator groups that have access to the analysis and reports provided by the task, and click OK.

¡     Baseline Analysis—Select Enable from the list.

The Enable Automatic Anomaly Detection Based On The Baseline parameter and the Baseline Threshold Setting area are displayed.

¡     Enable Automatic Anomaly Detection Based On The Baseline—Select Disable from the list.

¡     Threshold Alarm—Select Enable from the list.

The Threshold Alarm Settings area is displayed.

5.     In the Baseline Threshold Settings area, set the In Threshold to 40 and the Out Threshold to 30.

6.     In the Threshold Alarm Settings area, set the In Threshold to 50 Mbps and the Out Threshold to 30 Mbps.

7.     In the Interface Information area, click Select to select interface FortyGigE 1/0/2.

8.     Use the default settings for other parameters.

9.     Click OK.

Figure 5 Adding an interface traffic analysis task

 

Verifying the configuration

Verifying the configuration on the device

# Display NetStream entry information in the cache.

[Device] display ip netstream cache

IP NetStream cache information:

  Active flow timeout             : 5 min

  Inactive flow timeout           : 300 sec

  Max number of entries           : 1048576

  IP active flow entries          : 2

  MPLS active flow entries        : 0

  L2 active flow entries          : 0

  IPL2 active flow entries        : 0

  IP flow entries counted         : 0

  MPLS flow entries counted       : 0

  L2 flow entries counted         : 0

  IPL2 flow entries counted       : 0

  Last statistics resetting time  : Never

 

IP packet size distribution (11 packets in total):

 

 1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480

 .000 .000 .909 .000 .000 .090 .000 .000 .000 .000 .000 .000 .000 .000 .000

 

  512  544  576 1024 1536 2048 2560 3072 3584 4096 4608 >4608

 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000

 

 Protocol          Total  Packets    Flows  Packets Active(sec) Idle(sec)

                   Flows  /sec       /sec   /flow   /flow       /flow

---------------------------------------------------------------------------

 

Type DstIP(Port)            SrcIP(Port)            Pro ToS If(Direct)  Pkts

     DstMAC(VLAN)           SrcMAC(VLAN)

     TopLblType(IP/MASK)    Lbl-Exp-S-List

---------------------------------------------------------------------------

IP   10.1.1.1 (21)         100.1.1.2(1024)         1   0   FGE1/0/1(I) 5

IP   100.1.1.2 (1024)      10.1.1.1 (21)           1   0   FGE1/0/1(O) 5

# Display NetStream data export information.

[Device] display ip netstream export

IP export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 12.110.2.2 (5000)

  Version 5 exported flow number                  : 0

  Version 5 exported UDP datagram number (failed) : 0 (0)

  Version 9 exported flow number                  : 10

  Version 9 exported UDP datagram number (failed) : 10 (0)

Verifying the configuration on IMC

1.     View the summary reports for interface traffic analysis tasks.

a.     Click the Service tab.

b.     From the left navigation tree, select Traffic Analysis and Audit >  Interface Traffic Analysis Task.

The summary reports for interface traffic analysis tasks are displayed, as shown in Figure 6.

Figure 6 Summary reports for interface traffic analysis tasks

 

2.     View detailed reports for the interface traffic analysis task named Interface:

a.     Click the Service tab.

b.     From the left navigation tree, select Traffic Analysis and Audit >  Interface Traffic Analysis Task.

The page displaying the summary reports for all interface traffic analysis tasks are displayed.

c.     Use either of the following methods to access the report page of interface traffic analysis task Interface:

-     In the Summary List area, click the name of interface traffic analysis task.

-     On the left navigation tree, move your mouse pointer to the shortcut menu icon  next to Interface Traffic Analysis Task, and then select Interface from the menu.

The Traffic tab displays the traffic statistics reports of the task.

Figure 7 Viewing the traffic statistics reports of the task

 

3.     To view the application usage reports of the task, click the Application tab.

Figure 8 Viewing the application usage reports of the task

 

 

Example: Configuring NetStream aggregation data export

Network configuration

As shown in Figure 9, all routers in the network are running EBGP. Configure NetStream on the device to collect and export aggregated flow statistics as follows:

·     Export NetStream traditional data in version 5 format to the IMC server with IP address 4.1.1.1/16 and port 5000.

·     Perform NetStream aggregation in the modes of protocol-port, source-prefix, destination-prefix, and prefix.

·     Export the aggregated data of different modes to UDP ports 3000, 4000, 6000, and 7000 of the IMC server.

Figure 9 Network diagram

 

 

Configuring the device

# Assign an IP address to each interface, as shown in Figure 9. (Details not shown.)

# Specify version 5 format for NetStream traditional data export.

<Device> system-view

[Device] ip netstream export version 5 origin-as

# Enable NetStream for incoming and outgoing traffic on FortyGigE1/0/1.

[Device] interface fortygige 1/0/1

[Device-FortyGigE1/0/1] ip netstream inbound

[Device-FortyGigE1/0/1] ip netstream outbound

[Device-FortyGigE1/0/1] quit

# Specify 4.1.1.1 as the IP address of the destination host and UDP port 5000 as the destination port number.

[Device] ip netstream export host 4.1.1.1 5000

# Enable the protocol-port aggregation mode, and specify the destination host for the aggregation data export.

[Device] ip netstream aggregation protocol-port

[Device-ns-aggregation-protport] enable

[Device-ns-aggregation-protport] ip netstream export host 4.1.1.1 3000

[Device-ns-aggregation-protport] quit

# Enable the source-prefix aggregation mode, and specify the destination host for the aggregation data export.

[Device] ip netstream aggregation source-prefix

[Device-ns-aggregation-srcpre] enable

[Device-ns-aggregation-srcpre] ip netstream export host 4.1.1.1 4000

[Device-ns-aggregation-srcpre] quit

# Enable the destination-prefix aggregation mode, and specify the destination host for the aggregation data export.

[Device] ip netstream aggregation destination-prefix

[Device-ns-aggregation-dstpre] enable

[Device-ns-aggregation-dstpre] ip netstream export host 4.1.1.1 6000

[Device-ns-aggregation-dstpre] quit

# Enable the prefix aggregation mode, and specify the destination host for the aggregation data export.

[Device] ip netstream aggregation prefix

[Device-ns-aggregation-prefix] enable

[Device-ns-aggregation-prefix] ip netstream export host 4.1.1.1 7000

[Device-ns-aggregation-prefix] quit

Configuring IMC

Perform the configuration tasks on IMC as described in "Example: Configuring NetStream traditional data export."

Verifying the configuration

Verifying the configuration on the device

# Display NetStream data export information.

[Device] display ip netstream export

protocol-port aggregation export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 4.1.1.1 (3000)

  Version 8 exported flow number                  : 2

  Version 8 exported UDP datagram number (failed) : 2 (0)

  Version 9 exported flow number                  : 0

  Version 9 exported UDP datagram number (failed) : 0 (0)

 

source-prefix aggregation export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 4.1.1.1 (4000)

  Version 8 exported flow number                  : 2

  Version 8 exported UDP datagram number (failed) : 2 (0)

  Version 9 exported flow number                  : 0

  Version 9 exported UDP datagram number (failed) : 0 (0)

 

destination-prefix aggregation export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 4.1.1.1 (6000)

  Version 8 exported flow number                  : 2

  Version 8 exported UDP datagram number (failed) : 2 (0)

  Version 9 exported flow number                  : 0

  Version 9 exported UDP datagram number (failed) : 0 (0)

 

prefix aggregation export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 4.1.1.1 (7000)

  Version 8 exported flow number                  : 2

  Version 8 exported UDP datagram number (failed) : 2 (0)

  Version 9 exported flow number                  : 0

  Version 9 exported UDP datagram number (failed) : 0 (0)

 

IP export information:

  Flow source interface                           : Not specified

  Flow destination VPN instance                   : Not specified

  Flow destination IP address (UDP)               : 4.1.1.1 (5000)

  Version 5 exported flow number                  : 10

  Version 5 exported UDP datagram number (failed) : 10 (0)

  Version 9 exported flow number                  : 0

  Version 9 exported UDP datagram number (failed) : 0 (0)

Verifying the configuration in IMC

View the traffic analysis reports of the traffic analysis task in IMC. For operation procedures, see the "Verifying the configuration on IMC" section in "Example: Configuring NetStream traditional data export."