EventMusic Producer is a Dockerized application designed to read data and output them to a Kafka topic, using Avro schemas for data serialization. It integrates seamlessly with Kafka and the Schema Registry to manage the flow of event data linked to music event information.
Page view events (page_view_events) are generated when users browse the site. They are structured as follows:
{
"ts": "Timestamp of the event",
"sessionId": "Session identifier",
"page": "URL of the page visited",
"auth": "Authentication status (Logged In/Logged Out)",
"level": "Subscription type (free/paid)",
"itemInSession": "Number of items in the session",
"city": "User's city",
"zip": "User's zip code",
"state": "User's state/region",
"userAgent": "Browser user agent",
"lon": "Longitude",
"lat": "Latitude",
"userId": "User identifier",
"lastName": "User's last name",
"firstName": "User's first name",
"gender": "User's gender",
"registration": "User registration timestamp",
"artist": "Listened artist",
"song": "Song listened to",
"duration": "Song duration in seconds".
}
Listen events (listen_events) are generated when users listen to songs. Their structure is as follows:
{
"artist": "Artist name",
"song": "Song title",
"duration": "Song duration in seconds",
"ts": "Event timestamp",
"sessionId": "Session identifier",
"auth": "Authentication status",
"level": "Subscription type",
"itemInSession": "Number of items in session",
"city": "Ville",
"zip": "Zip code",
"state": "State/Region",
"country": "Pays",
"userAgent": "Browser user agent",
"lon": "Longitude",
"lat": "Latitude",
"userId": "User identifier",
"lastName": "Last name",
"firstName": "First name",
"gender": "Sexe",
"registration": "Registration timestamp".
}
Authentication events (auth_events) are generated when users attempt to log in. Structure :
{
"ts": "Timestamp of the event",
"sessionId": "Session identifier",
"level": "Subscription type",
"itemInSession": "Number of items in session",
"city": "Ville",
"zip": "Zip code",
"state": "State/Region",
"userAgent": "Browser user agent",
"lon": "Longitude",
"lat": "Latitude",
"userId": "User identifier",
"lastName": "Last name",
"firstName": "First name",
"gender": "Sexe",
"registration": "Registration timestamp",
"success": "Successful authentication (True/False)".
}
- Kafka Integration: Seamlessly produces music event data to Kafka topics.
- Avro Serialization: Utilizes Avro schemas for efficient data serialization and validation.
- Docker Support: Fully Dockerized for easy deployment and scaling.
- Batch Processing: Sends data in batches to optimize network use and processing time.
- Flexible Configuration: Easy to configure with environment variables for Kafka, Schema Registry, and more.
Follow these steps to get the application running with Docker:
docker pull stefen2020/eventmusic:latest
Make sure Kafka and Schema Registry are running and accessible. Replace localhost with your Kafka and Schema Registry hosts if necessary.
docker run --network="host" --name eventmusic-container stefen2020/eventmusic:latest
Make sure Kafka and Schema Registry are running and accessible. Replace localhost with your Kafka and Schema Registry hosts if necessary.
python listen_events.py
python page_view_events.py
python auth_events.py
python main.py
Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.
- Name: Stefen Taime
- Email: [email protected]
- Project URL: https://github.com/Stefen-Taime/eventmusic
- Project HUB DOCKER URL: https://hub.docker.com/r/stefen2020/eventmusic