DETECTION OF RANSOMWARE MALWARE IN THE WINDOWS OPERATING SYSTEM

Abstract

The rapid evolution of cyber threats has significantly increased the complexity and frequency of attacks against modern information systems, posing serious challenges to the security of governmental, corporate, and private infrastructures. Among the most destructive forms of malicious software, ransomware remains one of the most critical threats due to its ability to encrypt or block access to data and demand ransom for recovery. The widespread adoption of advanced evasion techniques—such as polymorphism, obfuscation, code injection, fileless execution, and the abuse of legitimate system processes—has considerably reduced the effectiveness of traditional signature-based detection mechanisms. Furthermore, the emergence of the Ransomware-as-a-Service (RaaS) model has accelerated the proliferation and accessibility of ransomware attacks, making them a persistent and large-scale cybersecurity problem. This paper addresses the problem of ransomware detection in the Windows operating system environment by proposing a hybrid detection model that integrates dynamic behavioral monitoring with machine learning techniques. The study begins with a comprehensive analysis of existing ransomware detection approaches, including signature-based, behavioral, and hybrid solutions. While signature-based methods offer high efficiency for known threats, they fail to detect novel or obfuscated malware variants. Behavioral approaches, on the other hand, demonstrate greater adaptability by analyzing runtime activities such as file system operations, system calls, memory usage, and network behavior, but they may suffer from increased false-positive rates. Hybrid models that combine static and dynamic analysis are therefore considered the most promising direction for achieving an optimal balance between detection accuracy and system performance. Based on this analysis, a hybrid ransomware detection framework is proposed that combines Random Forest (RF) and Support Vector Machine (SVM) classifiers with a radial basis function kernel. The model operates through continuous real-time monitoring of file system, system call, and network activities of running processes in the Windows OS. For each process, a behavioral feature vector is constructed, incorporating characteristics such as file write and rename frequency, entropy of modified files, read-to-write ratios, average time between events, and the number of unique outbound IP connections. All feature vectors are normalized to the [0,1] range before classification. The RF classifier is employed to identify typical ransomware behavior patterns based on ensemble decision trees, ensuring robustness and resistance to overfitting. Simultaneously, the SVM model focuses on detecting anomalous deviations in a high-dimensional feature space. The final classification decision is made using a weighted voting mechanism that balances the contributions of both models, allowing the system to maintain high sensitivity to malicious activity while reducing false alarms. If the computed probability of ransomware behavior exceeds a predefined threshold, the system initiates mitigation actions such as process blocking, network isolation, and event logging. The proposed architecture consists of five main components: a file system monitoring module, a system call analysis module, a network monitoring module, a feature extraction module, and a hybrid classification module. This modular design enables efficient real-time operation and scalability for deployment in both local and large-scale corporate environments. Unlike traditional static detection systems, the proposed model can identify previously unknown ransomware variants, including zero-day and fileless attacks, by detecting deviations from normal behavioral patterns during the early stages of execution. The results of the conceptual and architectural analysis demonstrate that the proposed hybrid approach is well-suited for real-time ransomware detection in Windows environments. It provides a high level of adaptability, computational efficiency, and detection accuracy while maintaining resilience against modern evasion techniques. Future research directions include expanding the behavioral feature set, integrating real-time network traffic analysis, incorporating automated model retraining mechanisms, and exploring the use of deep learning techniques to further enhance detection capabilities.