Hadoop Ecosystem Project

Description

This project involves setting up a Hadoop ecosystem to process logs received from Flume and NiFi. Batch processing is handled by Hadoop, while real-time processing is done via Kafka, Storm & Esper, and Redis. The data is then stored in a data warehouse using Hive and HUE and finally served using Java Spring.

Table of Contents

• Tech Stack

Tech Stack

Data Ingestion

• Flume
• NiFi

Batch Processing

• Hadoop

Real-time Processing

• Kafka
• Storm
• Redis

Data Warehousing

• Hive
• HUE
• PostgreSQL

Serving Layer

• Java Spring

programming language

• Java
• Python