Can Khiops be deployed on Hadoop?

[lucaurelien]

This question is inspired by inquiries we’ve received via our contact form from users interested in deploying Khiops in Hadoop environments.

[lucaurelien]

Yes, Khiops has been used for several years in production on Hadoop using specialized tools we have developed. However, these components are not yet part of its open-source distribution.

Overview of the technical solution

We have developed the following components to enable Khiops integration with Hadoop ecosystems:

• HDFS driver for Khiops: A dedicated driver enabling Khiops to read and write directly from HDFS, with full support for multipart files;
• Khiops4Spark: A Spark wrapper that enables Khiops to run on Hadoop clusters through spark-submit command;
• Khiops4Yarn: A Yarn wrapper that allows Khiops to operate directly within Hadoop’s resource management framework.

Khiops, as a native MPI (Message Passing Interface) program, works exactly the same on a multicore machine or on a multinode cluster. When deployed on multiple machines, Khiops efficiently leverages the CPU, memory, and I/O resources of each allocated node, ensuring optimal performance at scale. The Spark and Yarn wrappers manage the communication with their respective resource managers, which in turn allocate resources that will be used by Khiops.

If you’re interested in deploying Khiops in such environments or have specific use cases, please reach out via our contact form for further information.