P2P traffic detection in BWM version 2.9.1 vs 3.0.8

Started by Ronny

Ronny

P2P traffic detection in BWM version 2.9.1 vs 3.0.8   29 November 2014, 19:48

Hi Andrew,

I have noticed the following between the two versions: P2P detection in version 2.9.1 is very accurate and limits P2P traffic 100%. However in version 3.0.8 P2P traffic is rarely detected even with DPI using variants of rules and parameters.

Is there no way to incorporate Version 2.9.1's way of detecting P2P traffic into Version 3.0.8?

What I have noticed is, for example: I have blocked all IPv6 traffic during testing.

Version 2.9.1 (It limits P2P perfectly, and detects all other traffic with rule 2 correctly)

Rule 1:
Source: Any IP Address
Destination: Any IP Address
Protocol: P2P
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: Any IPv4 Based
Interface: LAN

Version 3.0.8 (when rule 1 does eventually match P2P and reaches the set limits, it overflows to rule 2, this basically does not limit P2P and P2P can run at full speed showing traffic on both rules, I have tried almost every setting and variant to the rules with no luck)

Rule 1:
Source: Any IP Address (also tried specifying local IP ranges, mac address, etc)
Destination: Any IP Address
Protocol: IPv4 TCP/UDP (tried all options)
DPI Matching: P2P
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 Based
Interface: LAN

I hope that makes sense as what I am trying to say.
SoftPerfect Support forum - Andrew avatar image

Re: P2P Traffic BWM 2.9.1 vs 3.0.8   29 November 2014, 22:17

In the 3.x branch we are using an open source library called NDPI which is a successor of OpenDPI.

NDPI looks like a quality library with a rich set of recognised protocols and it identified P2P and uTorrent on your tests pretty well. What application does it fail to catch for you? I would also recommend to try version 3.0.9 as it contains the latest version of NDPI.
Ronny

Re: P2P Traffic BWM 2.9.1 vs 3.0.8   30 November 2014, 06:01

Hi Andrew,

Thanks for the response, I can understand the DPI sometimes missing traffic. I have noticed mostly torrent traffic in general is not matched by the DPI. I personally use uTorrent, and the DPI never detects traffic from it, only if I use other torrent clients is traffic detected.

But the second point I made is more of a problem, where traffic is allowed to exceed the rate limiting, which does not occur in version 2.9.1. Version 3.0.8 shows in the live bandwidth monitor that torrents are running at the set rate limit, but on the test PC torrents are running way over the set rate limit. And traffic on DPI rules does not seem to get rate limited.

I tried the following in Version 3.0.8 on have a 40MBPS connection, I used a clean PC with all auto updates turned off and nothing running except for what I was testing:

Test 1
Rule 1:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 TCP/UDP
DPI Matching: P2P
Rate Limit: 32KB/s : 512KB/s
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 Based
Rate Limit: Unlimited
Interface: LAN

Live Monitor showed
Rule 1: Upload = 32KB/s Download = 512KB/s (Maxed out)
Rule 2: Upload = 201KB/s Download = 2300KB/s

Torrent client on PC showed Upload = 230KB/s Download = 2800KB/s

Test 2
Rule 1:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 TCP/UDP
DPI Matching: YouTube
Rate Limit: 16KB/s : 256KB/s
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 Based
Rate Limit: Unlimited
Interface: LAN

Live Monitor showed
Rule 1: Upload = 3KB/s Download = 256KB/s (Maxed out download)
Rule 2: Upload = 2KB/s Download = 1800KB/s

YouTube 'stats for nerds showed' while streaming a 1080p video Upload = 3KB/s Download = 2100KB/s

This was clearly showing that DPI was matching most times, but was NOT limiting bandwidth accordingly, and allowing a 'over flow' of traffic to the next rule once the previous rule had reached it's set rate limit. This is basically making the DPI useless, as BWM is NOT limiting bandwidth and everything gets full speed all the time regardless of rate limit set.
SoftPerfect Support forum - Andrew avatar image

Re: P2P traffic detection in BWM version 2.9.1 vs 3.0.8   08 December 2014, 15:59

Hi Ronny,
Thank you for the detailed information. I now see what the issue is, though I don't think it's actually rule over-flowing. It looks like some connections are identified by DPI and some are not.

Could you perhaps repeat the experiment with Rule 1 set to Unlimited and post the result here?
Hi Andrew,

I had a moment to do a test, I ran the same well seeded torrent using the two sets of rules with uTorrent.
This clearly shows that the DPI is accurately detecting Torrent traffic, but once it reaches the rate limit set on the rule it allows the traffic to over flow to the rule.
I have found the new version only rate limits traffic when NOT using DPI, The minute DPI is enabled, it is as if Rate Limiting is ignored.

Test 1
Rule 1:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 TCP/UDP
DPI Matching: P2P
Rate Limit: Unlimited
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 Based
Rate Limit: Unlimited
Interface: LAN

Live Monitor showed
Rule 1: Upload = 384KB/s Download = 3840KB/s (Maxed out line at line speed)
Rule 2: Upload = 3KB/s Download = 5KB/s

Test 2
Rule 1:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 TCP/UDP
DPI Matching: P2P
Rate Limit: 32KB/s : 512KB/s
Interface: LAN

Rule 2:
Source: Any IP Address
Destination: Any IP Address
Protocol: IPv4 Based
Rate Limit: Unlimited
Interface: LAN

Live Monitor showed
Rule 1: Upload = 32KB/s Download = 512KB/s (Maxed out Rule)
Rule 2: Upload = 290KB/s Download = 3105KB/s
SoftPerfect Support forum - Andrew avatar image

Re: P2P traffic detection in BWM version 2.9.1 vs 3.0.8   21 January 2015, 13:20

Thank you for providing these details. I delved into the issue and I found that the latest versions of μTorrent actively monitors its download/upload rates, as well as connection congestion.

When we don't set a limit, everything works okay and the P2P rule detects everything.

When we do set a limit, soon enough μTorrent realises that the connection has become congested. Then it appears to begin to make use of rate control and switch to µTP. Unfortunately this protocol is very complicated and BM cannot currently detect it. That's why the P2P rule seems to be overflown.

As a temporary workaround I suggest to try the Penalties tab, which can be used to restrict long and/or large downloads regardless of upper level protocols used.
Hi Andrew,

Any chance of getting some sort of way to shape/rate limit all P2P traffic?

I am still using version 2.9.1 because it catches all P2P traffic and limits it properly. I would like to move to the newest version but as above it does not effectively catch all P2P traffic.

Is there no way to implement the way version 2.9.1 performs P2P shaping into the new version? It is far more affective then the new version. Maybe a dual option to allow both the new implementation and the old implementation in a rule?
SoftPerfect Support forum - Andrew avatar image

Re: P2P traffic detection in BWM version 2.9.1 vs 3.0.8   18 April 2016, 09:30

Sorry, but we can't bring the old P2P matching code to the latest version. While it appears to work, it's prone to false positive detections where non-P2P traffic may be marked as such.

Having said that, depending on your goals the latest version's DPI-based penalty may be a good workaround. Basically if it sees at least one P2P packet in the stream, it will penalise the user for the specified time period. This works well, even if say 1/10th of P2P traffic is detected as such because the entire connection is slowed down until P2P packets are no longer seen.

SoftPerfect support forum
My problem is that the new version does NOT throttle/shape P2P correctly. P2P seems to be able to bypass the DPI and exceeds the rate limits imposed, basically rendering the DPI useless. So I will just stick to 2.9.1 then.
Hi Andrew,

I have now tried DPI in every way possible within the latest BWM. It seems nothing is detected correctly P2P, Dropbox, YouTube, SIP, Skype all can exceed the rate limits set. Everything else seems to work correctly with the exception of DPI.

I cannot see how after trying for hours on end I can't get DPI to work. I thought it might be windows so I tried Win 2003, 2008r2, 2012r2 even windows 8.1. Tried using two different PC's yet in Version 2.9.1 DPI works flawlessly.

Something is not right, DPI is broken. I can't believe no one else is experiencing the same problem as I am?
SoftPerfect Support forum - Andrew avatar image

Re: P2P traffic detection in BWM version 2.9.1 vs 3.0.8   21 April 2016, 15:20

It's not that simple, unfortunately:
  • It will detect some P2P, but not all, especially when a torrent app switches to µTP.
  • DropBox and Youtube are only detected when not encrypted with SSL.
  • Skype communications are also all encrypted and difficult to detect.
This makes the DPI engine reasonably useful when we need to detect the presence of a protocol (for example for penalties as I described above) but not as useful when you want to capture all of them.

At the moment we are using the open-source software called NDPI. There's also a commercial alternative called PACE, though I am not sure at the moment about its price and whether we can use it in BM.

Sometimes you can get the answer faster if you try the forum search and/or have a look at the software user manual to see if your question has already been answered.

Our forum rules are simple:

  • Be polite.
  • Do not spam.
  • If possible, check your spelling and grammar.

Author:

Subject

A brief and informative title for your message, approximately 4–8 words:

     

Spam prevention: please enter the following code in the input field below.

 **     **  ********  **     **  ********   **      ** 
  **   **   **         **   **   **     **  **  **  ** 
   ** **    **          ** **    **     **  **  **  ** 
    ***     ******       ***     **     **  **  **  ** 
   ** **    **          ** **    **     **  **  **  ** 
  **   **   **         **   **   **     **  **  **  ** 
 **     **  ********  **     **  ********    ***  ***  

Message: