Authors:
Lakshmi Narasimha Raju Mudunuri

Addresses:
Department of Information Services, Valero Energy Corporation, San Antonio, Texas, United States of America.

Abstract:

As digital information flows increase increasingly quickly, there is a need for strong methodologies to manage the high velocity of data information flows in a digital analytical ecosystem. This research proposes an adaptive data engineering paradigm that efficiently transforms raw data into useful knowledge as quickly as possible. The research is based on a proprietary dataset of 156 log instances from the telemetry and transactional systems of a cloud-native, distributed system. The system uses Apache Kafka for real-time ingestion, Apache Flink for stream processing, and a custom-built dynamic resource allocator to ensure system stability under different load conditions. The proposed approach can be upgraded to overcome the bottleneck by leveraging automatic schema evolution and intelligent partitioning strategies. The proposed approach can be further enhanced by automating schema evolution and adopting intelligent partitioning strategies. Empirical analysis shows that the resulting system achieves a significant increase in throughput compared to conventional static pipeline architectures. Additionally, by adding a feedback-driven optimising module, the system can adapt the priority of operations based on data volatility. The result of this research offers a scalable template for organisations aiming to maintain analytical integrity in a changing environment and to bridge the gap between raw data ingestion and high-level knowledge synthesis.

Keywords: Adaptive Engineering; Knowledge Transformation; Scalable Analytics; Stream Processing; Data Velocity; Digital Analytical Ecosystem; Apache Kafka; Automatic Schema Evolution.

Received on: 28/04/2025, Revised on: 15/07/2025, Accepted on: 16/08/2025, Published on: 29/06/2026

DOI: 10.69888/FTSCL.2026.000700

FMDB Transactions on Sustainable Computer Letters, 2026 Vol. 4 No. 2, Pages: 97-106

• Views : 53
• Downloads : 10

Download PDF