Scalable Data Engineering Architectures for Real-Time Financial Transaction Auditing

Even modest-sized organisations face intense regulatory pressure to provide instant assurance that high volumes of financial transactions satisfy their risk and compliance controls. Yet current approaches adopt a slow batch-cycle mentality, lacking observable data provenance, whose absence challenges trustworthiness, experiment outreach and risk-based prioritisation. This paper investigates scalable data engineering architectures that deliver audit assurance on a continuous basis in a manner that meets the needs of businesspeople and machine learning practitioners alike. Architecture-level choices, trade-offs and evaluation metrics guide a selection of the numerous available solutions in the Process and Production phases of the Data Engineering Life Cycle.

Data Observability refers to the dimension that focuses on quantifying and reporting the quality, reliability and trustworthiness of data within a platform. With increasing volumes of data being processed for insight and analysis, understanding how data quality is met and maintained is vital. Data observability enables organisations to systematically quantify these parameters, visualising baselines over time and providing alerting and reporting to allow ongoing adjustment to business-critical processes and systems. Data Provenance fills the evidential gap underneath data observability. Data provenance provides a record of the origin and history of each item of data throughout its life cycle, typically including details from how original data was captured, processing steps applied and how it was published to a risk or compliance assurance system. The absence of data provenance in a high-volume data environment reduces the trustworthiness of those systems. In risk-based approaches to data assurance this often leads auditors to perform extensive outward experiments, logging few internal checks; resourcing operations for seen risks instead of unseen assurances.