Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning

Authors:
Guoqiang Shu;Yating Hsu;David Lee
Affiliations:
Department of Computer Science and Engineering, the Ohio State University, Columbus, USA OH 43210;Department of Computer Science and Engineering, the Ohio State University, Columbus, USA OH 43210;Department of Computer Science and Engineering, the Ohio State University, Columbus, USA OH 43210
Venue:
FORTE '08 Proceedings of the 28th IFIP WG 6.1 international conference on Formal Techniques for Networked and Distributed Systems
Year:
2008

Citing 8
Cited 3

Learning regular sets from queries and counterexamples

Information and Computation
Black Box Checking

FORTE XII / PSTV XIX '99 Proceedings of the IFIP TC6 WG6.1 Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols (FORTE XII) and Protocol Specification, Testing and Verification (PSTV XIX)
Inside the Windows Security Push

IEEE Security and Privacy
Violating Assumptions with Fuzzing

IEEE Security and Privacy
DART: directed automated random testing

Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
On state reduction of incompletely specified finite state machines

Computers and Electrical Engineering
Testing Security Properties of Protocol Implementations - a Machine Learning Based Approach

ICDCS '07 Proceedings of the 27th International Conference on Distributed Computing Systems
Discoverer: automatic protocol reverse engineering from network traces

SS'07 Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium

Fuzzing with code fragments

Security'12 Proceedings of the 21st USENIX conference on Security symposium
Fuzzing the ActionScript virtual machine

Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security
A novel vulnerability detection method for ZigBee MAC layer

International Journal of Grid and Utility Computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Network-based fuzz testing has become an effective mechanism to ensure the security and reliability of communication protocol systems. However, fuzz testing is still conducted in an ad-hoc manner with considerable manual effort, which is mainly due to the unavailability of protocol model. In this paper we present our on-going work of developing an automated and measurable protocol fuzz testing approach that uses a formally synthesized approximate formal protocol specification to guide the testing process. We adopt the Finite State Machine protocol model and study two formal methods for protocol synthesis: an active black-box checking algorithm that has provable optimality and a passive trace minimization algorithm that is less accurate but much more efficient. We also present our preliminary results of using this method to implementations of the MSN instant messaging protocol: MSN clients Gaim (pidgin) and aMSN. Our testing reveals some serious reliability and security flaws by automatically crashing both of them.