ALGORITHMIC CORRECTION OF THE “BATCH EFFECT” IN THE ETL PIPELINE OF AN INTELLIGENT PLATFORM FOR PROCESSING HETEROGENEOUS BIOMEDICAL DATA

Abstract

The article examines the data preprocessing stage within the ETL pipeline of an intelligent platform for processing biomedical indicators that aggregates data from distributed sensor networks (IoT devices and medical monitors) and public repositories (MIMIC-III, PhysioNet). It is substantiated that a key architectural challenge in integrating heterogeneous data sources is the “batch effect” – a systematic shift in data distribution caused by technical factors (instrumental error, environmental conditions, and interlaboratory variability) rather than by the patient’s physiological condition. A mathematical correction model based on the ComBat algorithm is proposed. The model accounts for additive and multiplicative batch effects using an empirical Bayes approach, while a separate clinical covariate matrix (age, sex, BMI, and chronic diagnoses) is incorporated to preserve biologically meaningful variability. A comparative analysis of methods for eliminating the “batch effect” (Z-score normalization, quantile normalization, ComBat, denoising, and Batch Normalization) and methods for imputing missing values (Mean/Median, KNN, MICE, LOCF/NOCB, MissForest, and GAIN) is presented, along with the corresponding software libraries used for their implementation. An architectural solution is proposed that involves separating the ComBat correction module into an independent microservice at the data preprocessing stage, enabling the platform to dynamically adapt to new sensor types without retraining the core model.