Building a modern data platform based on the data lakehouse architecture and cloud-native ecosystem

In today’s Big Data world, organisations can gain a competitive edge by adopting data-driven decision-making. However, a modern data platform that is portable, resilient, and efficient is required to manage organisations’ data and support their growth. Furthermore, the change in the data management architectures has been accompanied by changes in storage formats, particularly open standard formats like Apache Hudi, Apache Iceberg, and Delta Lake. With many alternatives, organisations are unclear on how to combine these into an effective platform. Our work investigates capabilities provided by Kubernetes and other Cloud-Native software, using DataOps methodologies to build a generic data platform that follows the Data Lakehouse architecture. We define the data platform specification, architecture, and core components to build a proof of concept system. Moreover, we provide a clear implementation methodology by developing the core of the proposed platform, which are infrastructure (Kubernetes), ingestion and transport (Argo Workflows), storage (MinIO), and finally, query and processing (Dremio). We then conducted performance benchmarks using an industry-standard benchmark suite to compare cold/warm start scenarios and assess Dremio’s caching capabilities, demonstrating a 12% median enhancement of query duration with caching. This research creates a blueprint using DataOps, Kubernetes, and Cloud-Native ecosystem to build a resilient Big Data platform following the Data Lakehouse architecture, the base for Machine Learning (MLOps) and Artificial Intelligence (AIOps). Using an iterative approach, we architectured and implemented the core of the platform, which is composable and cloud-agnostic, avoiding vendor lock-in, which could run on any Cloud provider or on-premises. The initial benchmarking showed that the platform could efficiently handle millions of records, benefiting from Apache Iceberg features. This research creates a blueprint using DataOps, Kubernetes, and Cloud-Native ecosystem to build a resilient Big Data platform following the Data Lakehouse architecture, the base for Machine Learning (MLOps) and Artificial Intelligence (AIOps). Using an iterative approach, we architectured and implemented the core of the platform, which is composable and cloud-agnostic, avoiding vendor lock-in, which could run on any Cloud provider or on-premises. The initial benchmarking showed that the platform could efficiently handle millions of records, benefiting from Apache Iceberg features.