diff --git a/content/blogs/2023-08-16-datamesh.md b/content/blogs/2023-08-16-datamesh.md index dbed82b634..6d314c7255 100644 --- a/content/blogs/2023-08-16-datamesh.md +++ b/content/blogs/2023-08-16-datamesh.md @@ -1,5 +1,5 @@ --- -title: "How a Global Retailer Enabled Data Mesh at Scale with Kestra" +title: "How Leroy Merlin France Enabled Data Mesh at Scale with Kestra" description: "Dive into their transformative journey migrating from Apache Airflow to a scalable Data Mesh Architecture with Kestra." date: 2023-08-16T16:00:00 category: Solutions @@ -9,7 +9,7 @@ author: image: /blogs/2023-08-16-datamesh.jpg --- -In its transformation journey towards a cloud-based data infrastructure, a major retail company employing more than 100,000 people encountered significant challenges. At the time, they relied on a traditional on-premises data platform, using Teradata as its database, Talend for data integration, and then relied on global operations teams through service requests for scheduling using tools such as Dollar U and Automic Workload Automation. The team of 30 data engineers, organized by business domains, faced three major bottlenecks: +In its transformation journey towards a cloud-based data infrastructure, Leroy Merlin France (LMFR) a global retail company employing more than 24,000 people encountered significant challenges. At the time, they relied on a traditional on-premises data platform, using Teradata as its database, Talend for data integration, and then relied on global operations teams through service requests for scheduling using tools such as Dollar U and Automic Workload Automation. The team of 30 data engineers, organized by business domains, faced three major bottlenecks: - An **infrastructure bottleneck** that required a rapid migration to a serverless cloud architecture using Google Cloud with BigQuery as the central source for business intelligence, analytics and AI. @@ -17,14 +17,14 @@ In its transformation journey towards a cloud-based data infrastructure, a major - A **delivery and automation bottleneck** that required the adoption of CI/CD and DataOps principles to improve data operations -Initially, the company turned to Apache Airflow, but a pilot project exposed several limitations. Looking for a better solution, they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a Data Mesh Architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics. +Initially, LMFR turned to Apache Airflow, but a pilot project exposed several limitations. Looking for a better solution, they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a Data Mesh Architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics. -The company has experienced a 900% increase in data production over the past two years. After adopting Kestra, the company experienced significant improvements in scalability, speed, reliability, data processing efficiency, and reduced cost. +Leroy Merlin France has experienced a 900% increase in data production over the past two years. After adopting Kestra, they experienced significant improvements in scalability, speed, reliability, data processing efficiency, and reduced cost. ## Selecting the Right Data Orchestration Solution ## ### Why Apache Airflow failed in their organization -The company considered a managed Apache Airflow service through Cloud Composer as its primary data orchestration solution due to the popularity of this open-source project. However, upon applying Airflow to a pilot project, they encountered several major issues unacceptable for such an organization: +Leroy Merlin France considered a managed Apache Airflow service through Cloud Composer as its primary data orchestration solution due to the popularity of this open-source project. However, upon applying Airflow to a pilot project, they encountered several major issues unacceptable for such an organization: 1. **Complexity in Simple Tasks**: The creation of workflows, previously straightforward, turned into a time-consuming task. Designing and maintaining DAGs amplified the complexity, causing processing delays and even generating bottlenecks instead of eliminating them. @@ -132,7 +132,7 @@ Data transfer was set up via HTTPS directly to the Kestra API, reducing dependen --- ### Enhance Supply with Report Automation ### -To support charitable initiatives, each store allocates a portion of the margin to be donated to program-affiliated associations. The task to calculate these sums, inform store leaders, and provide payment details is managed by Kestra. +To support charitable initiatives, each store allocates a portion of the margin to be donated to program-affiliated associations. The task of calculating these sums, informing store leaders, and providing payment details is managed by Kestra. The data platform team used Kestra's capabilities to develop a dedicated workflow to handle this process. This workflow calculates the precise amounts to be allocated based on store performance metrics and other relevant data. The resulting sums are then prepared for distribution to the respective stores. @@ -146,7 +146,7 @@ Despite the intricacies of handling large volumes of data and coordinating perso ![metrics Kestra](/blogs/2023-08-16-datamesh/metrics.png) -Before Kestra integration, domain teams executed less than half a million tasks monthly. They used tools such as Talend, scheduled by DollarU or AWA, but moving toward cloud and scale processes was a significant bottleneck. +Before Kestra integration, domain teams executed less than half a million tasks monthly. Leroy Merlin France used tools such as Talend, scheduled by DollarU or AWA, but moving toward cloud and scale processes was a significant bottleneck. However, with the shift to a data mesh organization and Kestra's integration, their task management surged to over 5 million tasks monthly, which amounts to 75 days of processing every single day! @@ -159,7 +159,7 @@ That growth was not possible to accomplish with Apache Airflow. ## Next steps -Kestra effectively addressed this top retailer's initial needs and exceeded expectations by facilitating an unexpected yet highly beneficial result: the establishment of a data mesh architecture. +Kestra effectively addressed Leroy Merlin France's initial needs and exceeded expectations by facilitating an unexpected yet highly beneficial result: the establishment of a data mesh architecture. By implementing a data mesh, Kestra has empowered teams throughout the organization to independently manage and produce their own data pipelines. This not only promotes efficiency and reduces bottlenecks but also encourages ownership. Over a span of 18 months, the cumulative user base expanded by more than **900%**, totaling over **500 users**. These users transitioned from legacy tools, which supported only a limited set of executions and faltered at scale, to executing millions of tasks per month with Kestra, thereby generating significant value for their business. diff --git a/public/blogs/2023-08-16-datamesh.jpg b/public/blogs/2023-08-16-datamesh.jpg index 54d7d8e2a1..012403f805 100644 Binary files a/public/blogs/2023-08-16-datamesh.jpg and b/public/blogs/2023-08-16-datamesh.jpg differ diff --git a/public/blogs/2023-08-16-datamesh/metrics.png b/public/blogs/2023-08-16-datamesh/metrics.png index c85abd9770..c7bbf779f4 100644 Binary files a/public/blogs/2023-08-16-datamesh/metrics.png and b/public/blogs/2023-08-16-datamesh/metrics.png differ