Data Extractors
Provides the extraction layer between the providers and a destination db (here Postgres)
- N/A
-
Note: Currently the version dlt==1.5.0 works with Python 3.11 (not Python 3.13)
-
Git clone the project
- git clone https://github.com/FredGH/extrator.git
-
Ensure python3 is pointing to the 3.11
- Go to https://www.python.org/downloads/ and download Python 3.11.x
- Go to Visual Studio Code, select the Python Interpreter (Shift+Cmd+P) as Python 3.11.x
-
Create the Python Env
- rm -rf venv/
- In Visual Studio Vode -> Ctrl+Shift+P -> Select Interpreter -> Python 3.11.x
- python3.11 -m venv venv
- source ./venv/bin/activate
- In Visual Studio Vode -> Ctrl+Shift+P -> Select Interpreter -> (venv) Python 3.11.x
- [optional] check Python3 version -> python3 -> should show Python 3.11.x
-
Install Requirements:
- python3.11 -m pip install --upgrade pip
-
Build Package:
- pip3.11 install -U pip setuptools
- python3.11 setup.py sdist bdist_wheel
- pip3.11 install -e .
- [Optional]:
- In case you get this error, then follows the resolution steps:
- Error: "[...]Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. [...]"
- Resolution Steps:
- In the venv terminal:
- run curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
- . "$HOME/.cargo/env"
- pip3.11 install -e .
- In the venv terminal:
- In case you get this error, then follows the resolution steps:
-
Tag New Release & Push:
- git tag 0.0.x -m "Release details"
- git push origin 0.0.x
- git push
-
Install Package:
- Go to Settings > Developer Settings > Personal access tokens (classic) > Generate new token with note e.g. "Upload package"
- Ensure you check the write:packages scope to grant the necessary permissions.
- Get the generated token, e.g. "12345"
- Get your Github user name, e.g. "my_user_name"
- Install the private package using the following:
- Template:
- pip3.11 install git+https://github.com/{{ username }}/{{ package name }}.git@{{ tag/version }}
- [Optional] pip3.11 install git+https://{{ your access token }}@github.com/{{ username }}/{{ repository name}}.git@{{ tag/version }}#egg={{ package name }}
- Example:
- pip3.11 install git+https://github.com/{{ username }}/[email protected]
- [Optional] pip3.11 install git+https://[email protected]/{{ username }}/[email protected]#egg=extractor
- [Optional] Deleting tags:
- List tags: git tag -l
- Delete origin: git push origin :refs/tags/{{tagname}} (e.g. 0.01)
- Delete locally: git tag --delete {{tagname}} (e.g. 0.01)
-
Code cleanup & Standardisation
- ruff check . &isort . &black .
-
Run unit tests
- run in /dataproviders
- coverage run -m unittest discover
- coverage report -m (or coverage hml)
- run in /dataproviders
- N/A
- N/A
Contributors names and contact info [email protected]
- 0.0.1
- First version only containing stubs
- 0.0.2
- Initial Release
- There is no license.
Inspiration, code snippets, etc.