In our previous posts we have talked about duplication problem and possible solutions. We have explained how to set up automatic deduplication and hopefully this was helpful. Now we are going to go one step further and explain how to set up manual deduplication and why or when this could be helpful.
If you haven't had time to check out our past post on this subject, you can find them here:
Devices configuration and NetFlow deduplication Net Admins think about their data and if it is actually correct and deduplicated. This post gives a short explanation of this problem and how to overcome it. https://www.netvizura.com/blog/how-to-configure-devices-and-not-duplicate-netflow
How to solve duplicated NetFlow caused by multiple exporters This post will help you understand and solve exporters deduplication - what to do when same flow is exported by different devices. https://www.netvizura.com/blog/how-to-solve-duplicated-netflow-caused-by-multiple-exporters
In general, if you have correctly configured exporters (ingress/egress) and decided to enable automatic deduplication by exporting from all devices in flow continuity then all flows in your Traffic Patterns should be automatically deduplicated. However, if this is not the case then it is also possible for you to adjust Traffic Pattern configuration to achieve flow deduplication.
Deduplication based on the central exporter
If you have a central exporter (a netflow exporter through which all desired traffic is passing through) then preventing duplicated Traffic Pattern traffic is easy. You just need to add a filter to the Traffic Pattern in the Exporter section of the Traffic Pattern definition. Add the IP address of the central exporter while include option is set. This will result in Traffic Pattern matching only netflow that was exported by the central exporter.
In our example above, flow that passes and is exported by three routers (R1, R2 and R3) will be taken into account and processed only from central router (R2) since Traffic Pattern includes its IP address in Exporter filter.
Have in mind that all other traffic (passing via central exporter) will not be captured.
Deduplication based on exporters and their interfaces
If you do not have a central exporter and/or your network topology is more complex, you can prevent duplicated Traffic Patterns by entering exporters and their specific interfaces from which you will either include or exclude traffic, when matching traffic to a Traffic Pattern. That way you can exclude specific interfaces on exporters that would duplicate the traffic.
In the example above, flow travelling via R1 and R2 will not be duplicated since R2 is not an exporter, however flow travelling via R1 and R3 will be duplicated. By excluding Interface Out: Vl3 on Exporter R1 only export from exporter R3 will be processed.
Have in mind that all other traffic (via included exporters and interfaces) will be captured.
Deduplication based on next hop
In the example below, a flow travelling from Host A to Host B passes via two central routers R1 and R2. As a consequence, one flow is exported and processed to a netflow server twice (by R1 and R2). This should be overcome by adding next hop filter.
The solution is to exclude R2 as Next Hop IP address. This will simply skip all the flows passing from router R1 to R2. Flows will be then matched and processed only by router R2. The same applies for flows from Host B to Host A - excluding R1 as Next Hop will skip flows from R2 to R1.
Have in mind that all other traffic (not having R2 and R1 as next hop) will be captured.
Deduplication at router interfaces
Alternatively, you can avoid duplicated traffic even on routers themselves. It could be accomplished if you do not configure NetFlow on the interfaces which connect backbone routers.
In this case, you should disable deduplication in the in your NetFlow Analyzer application.
With so many solutions, now you can decide whether to use automatic deduplication or one of the manual options. Keep in mind notations that we have given above, so that you can be sure which traffic you are receiving.
