Acquiring Product Data from Nykaa, Blinkit, DMart, and Other Indian Retailers for Price Comparison Tool GSSoC'24

[shwetarkadam]

Problem Statement :
Problem Statement: Acquiring Product Data from Nykaa, Blinkit, DMart, and Other Indian Retailers for Price Comparison Tool
Objective:
Find out how we can get the product prices and details from leading Indian retailers like Nykaa, Blinkit, and DMart to facilitate price comparison for consumers.Should we access public apis,web scraping any other way?
The objective is to provide data to the backend and show that data on frontend.(this issue is focused on backend)

Challenges:

Data Acquisition: Identify and implement methods for obtaining product data, including prices, from retailers without public APIs. Possible methods include web scraping and exploring any available affiliate programs.
Legal and Ethical Compliance: Ensure all data collection techniques comply with the terms of service of the target websites and relevant legal guidelines.
Data Processing and Display: Develop processes to normalize and compare data from various sources and design a user interface that effectively displays this information to users.

Reference Docs:
https://www.nykaa.com/app-api/index.php/products/footer_categories?ptype=plp&id=2199

Level 3: 45 points

Bonus Level 2 points if someone shows their research and helps to breaks this problem into small easier problem

[Akhil-donthula-8977]

Hello @shwetarkadam, please assign me this task. I can contribute my best to this issue.Here is my linked profile: https://www.linkedin.com/in/akhil-donthula-63084626a/ and my portfolio [https://akhil-donthula.onrender.com/](url).
I believe adding filter options based on interests and enabling product comparison based on the filtered results would enhance the user experience. I am proficient in Next.js and also have some knowledge of Flask; however, I believe using Python for data processing could be a better option.

[palayushi293]

hello sir,, i have a good knowledge of web scraping , data science and flask pls assign this to me
Waiting for your response

[TohitCU]

I am knowledgeable in data analysis i can help you with it !
What i'm planning for it :
1)Reviewing Website Policies for data acquisition and legal and ethical complainces.

2)Check Robots.txt File if i'm allowed to scrape data from particular websites.

3)Monitor Changes to see any updates on the particular website's permissions.

[saikuladeepgithub]

Hi,
I have learnt web scraping. Recently I scraped the Amazon website. I think I am capable of doing this. Could you please assign me this

[kauxp]

Hi, I have previously worked on Python web scraping for websites like Amazon and Flipkart, for price comparisons of the products and I have also worked with open APIs like Swiggy. I think I can provide the best possible solution to this issue, so could you please assign this issue to me?

[khushikunte]

Hello ,I have Knowledge in this given section.I am intreseted in completing this task. Please Assign me this Under GSSOC'24

[Rv43]

Hello mam, i have a knowledge of website inspection, web scraping libraries and code, it's measures, data store and handling . Assign me thsi issue under gssoc

[JyotikaJayani-08]

Please assign this issue to me.
I am working on it.

[Himanshi11045]

Please assign this issue to me.

[Taranpreet10451]

Please assign this issue to me.

[Ar7109]

Hello @shwetarkadam ,

I'm eager to contribute to the Price Comparison Tool backend development task. With my experience in Next.js and proficiency in Python, I believe I can make significant contributions to this project.
envision enhancing user experience by incorporating filter options based on interests and enabling product comparison based on the filtered results. Leveraging Python for data processing seems to be the optimal choice, considering its efficiency and versatility.

I'm excited about the opportunity to work on this issue and contribute to the project's success. Please assign this task to me, and I'll ensure timely and high-quality delivery.

[bhanushri12]

@shwetarkadam I have worked on an issue like this and I plan on following the below steps:
Sure! Here's a refined version of the proposal focusing on the high-level plan and methodologies without the code.

Proposal: Acquiring Product Data from Nykaa, Blinkit, DMart, and Other Indian Retailers for Price Comparison Tool

Objective

The objective is to gather product prices and details from leading Indian retailers like Nykaa, Blinkit, and DMart to facilitate price comparison for consumers. This data will be provided to the backend and displayed on the frontend. The focus here is on backend data acquisition and processing.

Challenges

1. Data Acquisition:

   • Identify and implement methods for obtaining product data, including prices, from retailers without public APIs.
   • Possible methods include web scraping and exploring any available affiliate programs.

2. Legal and Ethical Compliance:

   • Ensure all data collection techniques comply with the terms of service of the target websites and relevant legal guidelines.

3. Data Processing and Display:

   • Develop processes to normalize and compare data from various sources.

Plan to Solve the Problem

1. Data Acquisition

• Web Scraping:
  • Using web scraping techniques to extract product data from retailer websites.
  • Handling dynamic content loading for websites that use JavaScript to load data.
• Public APIs:
  • public APIs provided by retailers for accessing product data.

2. Data Processing and Display

• Data Normalization:
  • Normalize data formats such as price formats and product names to facilitate accurate comparisons.
• Data Storage:
  • Store acquired data in a suitable database for easy access and manipulation.
• Backend Processing:
  • Implement backend logic for data retrieval, normalization, and storage using appropriate backend frameworks.
• Frontend Integration:
  • Develop a user-friendly interface to display the collected and processed data effectively.

Research Paper

For further reference and a deeper understanding of the methodologies and legal considerations involved in web scraping and data acquisition, refer to the research paper titled "Web Data Extraction, Applications and Techniques: A Survey".