Backend Architecture

[SwarnimWalavalkar]

Motivation

The goal of this architecture is to build a system with many loosely coupled, atomic components to facilitate a logically structured, extensible and modular system.

Components

Database Controller

Perform Database Operations

Data Aggregator

Aggregate data from multiple sources (Twitter, Telegram, OCR, etc), parse it, and call the database controller service to write to DB

Source Twitter

Fetch Tweets and pass it to the Data Aggregator to parse and write to DB

Source Telegram

Fetch data from telegram and send it to the aggregator service to parse

Source OCR

Use Google's Cloud Vision API to run OCR and send recognized text to the aggregator service

Supplier Load Balancer

Balance demand amongst suppliers to ensure no single supplier is sent as a response too many times in a short timeframe

Translator

Translate provided text using Google's Translate API, will be called by the aggregator service

Data Ingress Pipeline

[From Multiple Source Services (called by aggregator service)] → Data Aggregator (parsing and scheduling) → DB Controller

Data Egress Pipeline

[Consumer] → API → Database Controller → Supplier Load Balancer → Response

Assoicated Tasks/Goals

1. host Viksit’s code on our server and make some changes to the search queries (I will finalize the search queries for each resource type)
2. fetch tweets from v1.1 API using our bearer token
3. parse tweets to get contact number, resource type, city name and add to database (check for duplicates on contact number)
4. match the contact number with an evolving list of fraud numbers so that they get removed before adding to DB
5. have APIs which can deliver one contact at a time to the chatbot and take in response feedback regarding that lead
6. logic to load balance supply contacts between different requests to ensure that supplier doesn’t get flooded with inquiries
7. logic for scoring the leads in our database based on the inputs and dynamically rank them for each city + resource combo. check this discussion for some ideas