INTELLIGENT METHOD FOR DETECTING BUFFER OVERFLOW VULNERABILITIES IN SOFTWARE

Abstract

The rapid growth of information technologies and the complexity of software systems has intensified the risks of security vulnerabilities, among which buffer overflow remains one of the most critical. Traditional methods such as manual code auditing, static and dynamic analysis, and conventional testing are limited by scalability, accuracy, and high false-positive rates. This research proposes an intelligent method for detecting buffer overflow vulnerabilities that integrates mathematical modelling, symbolic execution, and reinforcement learning through Q-learning. The approach models buffer overflow conditions formally, employs symbolic variables to explore execution paths systematically, and applies Q-learning to mitigate the path explosion problem by prioritising the most promising branches. Constraint-based test input generation ensures realistic scenarios that activate potential vulnerabilities while reducing false positives. The method was implemented using LLVM-based symbolic execution (KLEE), dynamic instrumentation (Valgrind), and Python-based Q-learning with TensorFlow and OpenAI Gym. Experimental validation on synthetic benchmarks and real-world vulnerable code demonstrated improvements of over 30% in execution path coverage and a 20% reduction in false positives compared to classical symbolic execution. The study highlights the potential of combining formal models, symbolic analysis, and reinforcement learning to improve both precision and reliability in software vulnerability detection. The developed approach shows promise for integration into modern cybersecurity tools to enhance early identification of critical defects. Future work will focus on optimising computational efficiency, extending applicability to other vulnerability classes, and validating performance in large-scale industrial systems.