Flow Export Protocols in IP Networks

Traditional IP network management tools use the SNMP protocol to measure various performance parameters, such as error counters in interfaces and traffic volume (essential to the management of those networks). However, simply managing the infrastructure from the performance point of view is not enough, because as the traffic volume increases exponentially, network administrators must be intimately familiar with the details of that traffic. Such details are essential to understanding the behavior of applications, users, company offices and other individuals and structures that depend on those networks.

There was an attempt to provide a greater level of detail on the traffic of IP networks with the use of the SNMP protocol via the creation of MIB RMON – remote monitoring. However, it proved ineffective because it did not offer enough information on IP traffic and demanded enormous sophistication in the managing applications in order for them to be able to collect the data available in MIB RMON.

It soon became apparent that the SNMP protocol would not be capable of providing the necessary detail and that different approaches should be adopted. Flow export protocols proved themselves to be a more scalable way for acquiring essential information about network traffic

History

The first flow export technology appeared in 1996 when Cisco developed and patented NetFlow, a technology for optimizing the routing of packets in routers using extensive access control lists, or ACLs.

When NetFlow is enabled in the router, a table is created to maintain a cache of the routing decisions. After completing all of the calculations necessary to route a packet, the router stores the results in the table, associating them with a signature for the packet that was just routed. Subsequent packets arriving at the router with the same signature are routed via that table, without requiring the router to repeat all of those same calculations multiple times.

Later on, more efficient ways of routing packets were developed, but the information contained in the NetFlow table was extremely valuable for managing IP networks. As a result, NetFlow essentially became a technology used for that purpose.

NetFlow technology is supported by nearly the entire line of Cisco equipment. Different manufacturers followed suit and implemented similar or even identical technologies. Today the major manufacturers such as Juniper, Enterasys and Huawei all implement flow export.

Concept

A flow record represents a sequence of packets with the same signature in an IP network. That signature contains the following information:

• source IP;
• destination IP;
• TCP/UDP source port;
• TCP/UDP destination port;
• input interface of the network element;
• output interface of the network element;
• OSI Model Layer 4 Protocol (TCP, UDP, ICMP, etc.);
• class of Service represented by ToS (type of service) byte in the IP header.

IP packets sharing those attributes are grouped in a flow record, which provides packet and byte counters, flow start (first packet) and end (last packet or timeout) timestamps, autonomous systems for the source and destination, etc.

This information can be provided by the flow export protocols implemented by different manufacturers in addition to Cisco, Juniper, Enterasys and Huawei.

Main Applications

This information serves as the basis for resolving various problems common to IP networks, such as the impacts of applications on the network, identification of the biggest offenders (departments, locations, and users), identification of unauthorized traffic, cost assignment by departments, etc.

Most Common Traffic Analysis Practice

In many companies, traffic analysis is still carried out using either protocol analyzers or traffic probes scattered throughout the network. Each of these approaches has drawbacks.

The use of probes is not scalable. That is because in order to have a complete view of the network traffic with full-mesh technologies such as MPLS, it is necessary to have a probe for each location, making the solution very costly.

The protocol analyzer does not provide a real-time analysis. In other words, by the time the tool is being used, the problem is no longer happening.

Advantages of Flow Technology

Flow technologies permit continuous real-time analysis, because once enabled in routers, they continuously report on the traffic passing through them.

They are highly scalable, since routers or even Layer 3 switches can be converted into traffic probes, facilitating analysis of the entire LAN and WAN environment without adding new assets to the network structure.

The unique limiting factor of these technologies is that they do not provide information on the contents of the IP packets. This typically only allows for the identification of problems intrinsic to applications, such as the faulty implementation of the client-server communication protocol. Such problems should be identified with protocol analyzers and fall within the scope of application administrators and not network administrators.

Harnessing the Data

It turns out that merely receiving the flows from network elements capable of exporting it does not contribute much to an understanding of the traffic of an IP infrastructure due to the large number of records generated. So, it is very difficult to make the most of the technology without tools capable of organizing the data in a more effective manner.

Most of the oldest flow analysis tools currently available on the market do not take full advantage of the received flows. The most common analyses are of the applications in the traffic of interfaces and the detailed view of the flows, with recursive filters.

Today, we find some tools that are capable of carrying out a more flexible analysis and furnishing information that suits the client’s needs. They go beyond storing historic data for long periods, allowing for the analysis of trends or the making of projections for capacity planning.

Those tools support the creation of objects for analysis based on information contained in flow records and can work across those parameters to analyze the traffic of an object classified by another object. They make possible, for example, the analysis of the traffic of an application (such as SAP) classified by source or destination. In this way, the network administrator can see the impact of each region, site or department on the company’s network due strictly to SAP. This knowledge makes it possible to allocate the share of the network cost associated with SAP by region or usage group.

Another major resource offered by good flow analysis tools is the option to select and group similar traffic for a combined analysis. Imagine, for example, that a company’s marketing department is scattered throughout three sites and that those sites also house other departments sharing the network. The traditional tools do not identify only the usage of the marketing department, since they are based on SNMP and analyze the traffic of the link as a whole. A good flow analysis tool is capable of separating the specific traffic for one area (based on the IP of the hosts serving that area) and then classifying that traffic by application or any other criteria, independent of the physical link that was used. That provides an x-ray view of the usage and importance of the network for a given area of the company.

The tools can even pinpoint unauthorized or unanticipated uses of the network, showing a clear identification of the improper use.

Analyses such as those shown below are simple and rapidly obtained using those tools without the need for physical alterations to the network:

• Internet traffic usage by department and branch, facilitating resource usage accounting.
• Application server traffic by locations or departments, allowing the IT department to understand the usage pattern of key business applications and putting an end to the typical discussions between application and network administrators. With clear numbers, the teams can focus on solving the problem, whether by optimizing the application or expanding the network.
• Classification and detailing of the VoIP traffic, showing the impact of that application on the network and the biggest users.

Those analyses will be the basis for various actions, ranging from identifying key business applications and their needs for growth to supporting investment decisions for network expansion to uncovering traffic patterns related to viruses or misuse, identifying the machines involved in order to take appropriate action.

Conclusion

With a powerful tool for characterizing traffic, the network administrator can assign a name and number to each of its users. That tool is a tremendous ally when justifying the cost or need for investment in the network infrastructure, giving transparency to the company’s management and a full understanding of the importance of the network for each segment of its business. Thus, the network ceases to be an expensive black box.