Dynamic application-layer protocol analysis for network intrusion detection

  • Authors:
  • Holger Dreger;Anja Feldmann;Michael Mai;Vern Paxson;Robin Sommer

  • Affiliations:
  • TU München;TU München;TU München;ICSI/LBNL;ICSI

  • Venue:
  • USENIX-SS'06 Proceedings of the 15th conference on USENIX Security Symposium - Volume 15
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

Many network intrusion detection systems (NIDS) rely on protocol-specific analyzers to extract the higher-level semantic context from a traffic stream. To select the correct kind of analysis, traditional systems exclusively depend on well-known port numbers. However, based on our experience, increasingly significant portions of today's traffic are not classifiable by such a scheme. Yet for a NIDS, this traffic is very interesting, as a primary reason for not using a standard port is to evade security and policy enforcement monitoring. In this paper, we discuss the design and implementation of a NIDS extension to perform dynamic application-layer protocol analysis. For each connection, the system first identifies potential protocols in use and then activates appropriate analyzers to verify the decision and extract higher-level semantics. We demonstrate the power of our enhancement with three examples: reliable detection of applications not using their standard ports, payload inspection of FTP data transfers, and detection of IRC-based botnet clients and servers. Prototypes of our system currently run at the border of three large-scale operational networks. Due to its success, the bot-detection is already integrated into a dynamic inline blocking of production traffic at one of the sites.