Real-Time Detection of Automated Crawlers in High-Traffic Web Environments: An Integrated Approach

Abstract

Automated crawlers are widely used for legitimate Web services but can also generate resource-intensive traffic, distort analytics, and evade simple rule-based filtering in high-traffic environments. Existing approaches commonly rely on static fingerprints, aggregated session statistics, or computationally heavier sequence models, while confidence calibration and workload-dependent decision behavior are often evaluated separately. This study proposes a Load-Aware Integrated Crawler Detector (LAICD) that combines server-side session statistics with a lightweight temporal convolutional encoder, adaptive multi-view fusion, confidence calibration, and a bounded load-aware threshold controller. Evaluation on the held-out Phase 2 Web Bot Detection Dataset showed that LAICD achieved a Macro-F1 of 0.914 ± 0.008 and PR-AUC of 0.953 ± 0.006, compared with 0.892 ± 0.009 and 0.936 ± 0.008 for the strongest BiLSTM baseline. FPR@95TPR decreased from 8.8 ± 0.9% to 6.8 ± 0.7%. Removing confidence calibration increased ECE from 0.031 ± 0.006 to 0.071 ± 0.010. Under EClog replay, p95 inference latency increased from 2.8 ± 0.2 ms at 1× load to 6.2 ± 0.7 ms at 5× load. These results indicate that integrating temporal evidence, calibrated probabilities, and bounded workload adaptation can improve crawler detection while retaining practical runtime behavior and reducing dependence on additional client-side behavioral data.