METHOD FOR DETECTING MALICIOUS ACTIVITY IN IPSEC CHANNELS WITHOUT TRAFFIC DECRYPTION

Abstract

The paper addresses the problem of detecting malicious activity in encrypted communication channels without decrypting traffic contents. The relevance of the study is determined by the rapid growth of encrypted traffic in modern networks, where protocols such as HTTPS, QUIC, and VPN-based communication significantly reduce the applicability of traditional payload-based inspection and signature matching. Under such conditions, passive monitoring methods must rely on observable indirect features rather than packet contents.

The paper reviews current approaches to encrypted traffic analysis, including TLS fingerprinting methods, traffic classification based on packet length distributions, and practical monitoring solutions used in modern network security systems. It is shown that existing approaches achieve promising results in identifying traffic types in TLS sessions and VPN channels, but many of them depend on machine learning or deep learning models, exact fingerprint matching, or computationally expensive analysis pipelines. As a result, the problem of combining low computational complexity, real-time applicability, and sufficient detection capability remains open.

To address this issue, the paper proposes a method for identifying traffic types in encrypted IPsec channels based on the analysis of packet length distributions over short time intervals. For each interval, packet lengths observed in both transmission directions are aggregated and analyzed as a normalized distribution. Characteristic patterns are considered informative, including stable packet lengths in one direction, paired response/request patterns, multiple persistent packet-length lines, upper bounds of packet size, and wide packet-length spectra typical of file transfer. A schematic comparison of such patterns is provided for VoIP, FTP, and RDP traffic.

A key contribution of the paper is the use of the formal relationship between the observed ESP packet length and the length of the encapsulated payload. The total ESP packet length is represented as a sum of the payload size, service headers, encapsulation parameters, and padding. This makes it possible, for a known or estimated IPsec configuration, to calculate admissible ranges of encapsulated payload lengths and to use them during traffic classification. On this basis, an algorithm for determining traffic types inside an IPsec channel is proposed. The algorithm includes estimation of channel parameters, construction of packet-length distributions for both directions, calculation of admissible payload ranges, comparison with characteristic protocol patterns, and formation of a conclusion about the presence of particular traffic types.

The paper also considers practical issues of applying the proposed method. It is shown that the effectiveness of detection strongly depends on the configuration of the encrypted channel; therefore, partial models and reference distributions tailored to specific encapsulation settings are preferable to universal models. To reduce the cost of dataset preparation, the paper suggests constructing reference packet-length distributions from unencrypted traffic samples with subsequent adjustment for encapsulation headers and tunnel parameters. Another important aspect is the analysis of mixed traffic. Since multiple traffic types may coexist in the same encrypted channel, the paper proposes iterative analysis with predictive subtraction of already identified traffic patterns, especially for periodic traffic such as voice or video streams. This facilitates the detection of additional protocols that are initially masked by dominant flows.

Finally, a practical implementation concept is outlined in the form of multiple traffic sensors coordinated by a central orchestrator. Each sensor is responsible for detecting a specific class of traffic within a time window, while the orchestrator estimates and refines encrypted channel parameters and distributes them across the analysis modules. The proposed approach does not require payload decryption or neural networks, which reduces computational costs and makes it suitable for deployment in passive real-time traffic monitoring systems.