GitHub - paul7Junior/packet_sniffer_kafka_spark: scapy-kafka-spark-mongodb live stream packet analyser

paul7Junior / packet_sniffer_kafka_spark Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

scapy-kafka-spark-mongodb live stream packet analyser

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
misc		misc
README.Rmd		README.Rmd
README.nb.html		README.nb.html
pcaps-scappy-kafka-spark-mongodb.nb.html		pcaps-scappy-kafka-spark-mongodb.nb.html
sniffer.py		sniffer.py
spark.py		spark.py
spark_socket_stream.py		spark_socket_stream.py

Repository files navigation

---
title: "R Notebook"
output: html_notebook
---

Goal/Challenge: Being able to stream collect, process and store internet packets.

Tech stack: Scapy - Kafka - Spark - MongoDB - Docker

- Having all the services Dockerized
- Make it scalable with docker Swarm



How about being able to live stream internet packet with a sniffer and process it with Spark?

1. Get packet with Scapy sniffer.

2. Get a Kafka topic up and running.

3. Feed Kafka with scapy packets

4. Get Spark to consume kafka feeds

5. Do some simple operations on packets

6. Store processed data to mongoDB