Skip to content

Commit

Permalink
Update 2023-08-16-datamesh.md (#471)
Browse files Browse the repository at this point in the history
* Update 2023-08-16-datamesh.md

* Update 2023-08-16-datamesh.jpg

* Update 2023-08-16-datamesh.md

* Update metrics.png
  • Loading branch information
MartinRst authored Aug 29, 2023
1 parent e128865 commit f65b1c9
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions content/blogs/2023-08-16-datamesh.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "How a Global Retailer Enabled Data Mesh at Scale with Kestra"
title: "How Leroy Merlin France Enabled Data Mesh at Scale with Kestra"
description: "Dive into their transformative journey migrating from Apache Airflow to a scalable Data Mesh Architecture with Kestra."
date: 2023-08-16T16:00:00
category: Solutions
Expand All @@ -9,22 +9,22 @@ author:
image: /blogs/2023-08-16-datamesh.jpg
---

In its transformation journey towards a cloud-based data infrastructure, a major retail company employing more than 100,000 people encountered significant challenges. At the time, they relied on a traditional on-premises data platform, using Teradata as its database, Talend for data integration, and then relied on global operations teams through service requests for scheduling using tools such as Dollar U and Automic Workload Automation. The team of 30 data engineers, organized by business domains, faced three major bottlenecks:
In its transformation journey towards a cloud-based data infrastructure, Leroy Merlin France (LMFR) a global retail company employing more than 24,000 people encountered significant challenges. At the time, they relied on a traditional on-premises data platform, using Teradata as its database, Talend for data integration, and then relied on global operations teams through service requests for scheduling using tools such as Dollar U and Automic Workload Automation. The team of 30 data engineers, organized by business domains, faced three major bottlenecks:

- An **infrastructure bottleneck** that required a rapid migration to a serverless cloud architecture using Google Cloud with BigQuery as the central source for business intelligence, analytics and AI.

- A **data pipeline bottleneck** that required re-architecting the bottleneck of a central data team to a decentralized data integration lead directly from the product team.

- A **delivery and automation bottleneck** that required the adoption of CI/CD and DataOps principles to improve data operations

Initially, the company turned to Apache Airflow, but a pilot project exposed several limitations. Looking for a better solution, they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a Data Mesh Architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics.
Initially, LMFR turned to Apache Airflow, but a pilot project exposed several limitations. Looking for a better solution, they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a Data Mesh Architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics.

The company has experienced a 900% increase in data production over the past two years. After adopting Kestra, the company experienced significant improvements in scalability, speed, reliability, data processing efficiency, and reduced cost.
Leroy Merlin France has experienced a 900% increase in data production over the past two years. After adopting Kestra, they experienced significant improvements in scalability, speed, reliability, data processing efficiency, and reduced cost.

## Selecting the Right Data Orchestration Solution ##

### Why Apache Airflow failed in their organization
The company considered a managed Apache Airflow service through Cloud Composer as its primary data orchestration solution due to the popularity of this open-source project. However, upon applying Airflow to a pilot project, they encountered several major issues unacceptable for such an organization:
Leroy Merlin France considered a managed Apache Airflow service through Cloud Composer as its primary data orchestration solution due to the popularity of this open-source project. However, upon applying Airflow to a pilot project, they encountered several major issues unacceptable for such an organization:

1. **Complexity in Simple Tasks**: The creation of workflows, previously straightforward, turned into a time-consuming task. Designing and maintaining DAGs amplified the complexity, causing processing delays and even generating bottlenecks instead of eliminating them.

Expand Down Expand Up @@ -132,7 +132,7 @@ Data transfer was set up via HTTPS directly to the Kestra API, reducing dependen
---

### Enhance Supply with Report Automation ###
To support charitable initiatives, each store allocates a portion of the margin to be donated to program-affiliated associations. The task to calculate these sums, inform store leaders, and provide payment details is managed by Kestra.
To support charitable initiatives, each store allocates a portion of the margin to be donated to program-affiliated associations. The task of calculating these sums, informing store leaders, and providing payment details is managed by Kestra.

The data platform team used Kestra's capabilities to develop a dedicated workflow to handle this process. This workflow calculates the precise amounts to be allocated based on store performance metrics and other relevant data. The resulting sums are then prepared for distribution to the respective stores.

Expand All @@ -146,7 +146,7 @@ Despite the intricacies of handling large volumes of data and coordinating perso

![metrics Kestra](/blogs/2023-08-16-datamesh/metrics.png)

Before Kestra integration, domain teams executed less than half a million tasks monthly. They used tools such as Talend, scheduled by DollarU or AWA, but moving toward cloud and scale processes was a significant bottleneck.
Before Kestra integration, domain teams executed less than half a million tasks monthly. Leroy Merlin France used tools such as Talend, scheduled by DollarU or AWA, but moving toward cloud and scale processes was a significant bottleneck.

However, with the shift to a data mesh organization and Kestra's integration, their task management surged to over 5 million tasks monthly, which amounts to 75 days of processing every single day!

Expand All @@ -159,7 +159,7 @@ That growth was not possible to accomplish with Apache Airflow.

## Next steps

Kestra effectively addressed this top retailer's initial needs and exceeded expectations by facilitating an unexpected yet highly beneficial result: the establishment of a data mesh architecture.
Kestra effectively addressed Leroy Merlin France's initial needs and exceeded expectations by facilitating an unexpected yet highly beneficial result: the establishment of a data mesh architecture.

By implementing a data mesh, Kestra has empowered teams throughout the organization to independently manage and produce their own data pipelines. This not only promotes efficiency and reduces bottlenecks but also encourages ownership. Over a span of 18 months, the cumulative user base expanded by more than **900%**, totaling over **500 users**. These users transitioned from legacy tools, which supported only a limited set of executions and faltered at scale, to executing millions of tasks per month with Kestra, thereby generating significant value for their business.

Expand Down
Binary file modified public/blogs/2023-08-16-datamesh.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified public/blogs/2023-08-16-datamesh/metrics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

1 comment on commit f65b1c9

@vercel
Copy link

@vercel vercel bot commented on f65b1c9 Aug 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

kestra-io – ./

kestra-io-git-main-kestra.vercel.app
kestra-io.vercel.app
kestra-io-kestra.vercel.app

Please sign in to comment.