There are two options:
1) You can use the built-in DPI module that detects P2P activity. It uses heuristics and there is no guarantee it will always catch BitTorrent, but you can give it a go.
In a rule, to the
Advanced page and click
Even more advanced settings
2) Alternatively you can use
Penalties that allow you to discourage heavy downloads regardless of protocols used.