doc: add readme.md files with project information and instructions

README.md

@@ -0,0 +1,49 @@
+# Lenscan
+
+This project is aimed at offering an intuitive mechanism to explore the Lens Protocol. It consists of two main components: `bq-syncer` and `dashboard`.
+
+## Components
+
+- `bq-syncer`: This directory contains tools for syncing Lens BigQuery data to local DuckDB or parquet files.
+- `dashboard`: This includes a web application designed to render and display the processed data in an interactive way.
+
+## Getting Started
+
+### Dependencies
+
+Ensure you install and satisfy all the dependencies in each directory (bq-syncer and dashboard) before attempting to run this project. Please refer to each directory's respective README for detailed instructions.
+
+## Installation
+
+```sh
+git clone https://github.com/daoleno/lenscan.git
+cd lenscan
+```
+
+Then navigate to each directory and follow the instructions given in their respective README files.
+
+## Contribution
+
+We really appreciate the open-source community and their contributions. If you're keen to contribute, here are steps you can follow:
+
+1. Fork the repository.
+2. Make feasible changes to fix bugs or contribute with a new feature.
+3. Clean up your code and ensure the code style matches the style of the project.
+4. Generate a pull request.
+
+## License
+
+This project is licensed under the MIT License. Please see the [LICENSE](https://choosealicense.com/licenses/mit/) for details.
+About README
+
+## Authors & Acknowledgment
+
+• [daoleno](https://github.com/daoleno)
+
+Feel free to add your name here if you have contributed to the project.
+
+## Support
+
+If you like what you see here, please leave a star on Github and share it within your network. We appreciate your support!
+
+You can join our [Telegram](https://t.me/+Zq8XF9cI6GhkMTNl) to get in touch with the team and community.

bq-syncer/README.md

@@ -18,6 +18,19 @@ Before running the script, make sure you have downloaded the service account JSO
 ## Running the Script
 
 Run the script using a Python interpreter:
+Export all tables into parquet directory:
+
+```sh
+python sync_parquet.py -o  test/
+
+# sync a specific table
+python sync_parquet.py -o  test/ -t v2_polygon
+
+# concurrent sync
+python sync_parquet.py -o  test/ -c 4
+```
+
+Export all tables into a single DuckDB database and parquet directory:
 
 ```sh
 poetry shell
@@ -33,7 +46,7 @@ python sample.py --db /tmp/v2_polygon.db
 
 ## How it Works
 
-- On script start and every 15 minutes after, the script checks for new or updated rows in each table of the specified BigQuery dataset.
+- On script start and at every scheduled interval, the script checks for new or updated rows in each table of the specified BigQuery dataset.
 - Checks whether each table exists in the DuckDB database, creating it if not.
 - Retrieves the BigQuery table schema, converts it from RECORD to individual fields and creates the table in DuckDB with this schema if it does not already exist.
 - Retrieves the maximum `source_timestamp` value from the DuckDB table and fetches all rows from the BigQuery table that have a `source_timestamp` greater than this.