[FEATURE] Add transformation to download file from url #143

dannymeijer · 2024-12-02T10:21:24Z

Description

This pull request introduces a new feature to add a transformation for downloading files from a URL.

Related Issue

#75

Motivation and Context

N/A

How Has This Been Tested?

Unit tests were added

Screenshots (if appropriate):

N/A

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

…ransformationWithTarget and improve file download handling

riccamini

Some comments on the control flow of the DownloadFile step but overall nice addition 💪

src/koheesio/steps/download_file.py

… and adding type hints in tests

…s) + added additional tests + documentation updates

docs/releases/RELEASE_NOTES.md

src/koheesio/asyncio/http.py

src/koheesio/integrations/spark/databricks/secrets.py

src/koheesio/integrations/spark/tableau/hyper.py

src/koheesio/spark/readers/file_loader.py

src/koheesio/spark/readers/rest_api.py

src/koheesio/spark/utils/common.py

…and enhance test type hints

…ion with usage examples and write modes

… detailed write mode descriptions and examples

riccamini

Thanks for addressing the comments, looks food 💪

src/koheesio/spark/transformations/download_files.py

…h usage examples for transform and execute methods

dannymeijer added 4 commits December 2, 2024 10:19

added DownloadFileStep

163ebb8

remove redundant print

b21407a

made request method into contextmanager

cc40c4c

work in progress

55b24d3

dannymeijer linked an issue Dec 2, 2024 that may be closed by this pull request

[FEATURE] Add transformation to download file from url #75

Closed

dannymeijer and others added 11 commits December 2, 2024 11:42

work in progress

7660183

fixed failing test

c031ae8

work in progress

4cec1ab

work in progress

08b73ba

work in progress

c7bd346

kinda working now

91559f1

hotfix: refactor download file transformation and clean up unused code

b22fe82

refactor: update method reference in download file transformation

1f97416

added unit test

be1e394

small fix

cb2a369

work in progress

4b86809

dannymeijer added this to the 0.10.0 milestone Dec 9, 2024

dannymeijer added 3 commits December 10, 2024 11:22

Make request method private

bb5dd14

refactor: replace request method calls with private _request method

d623394

refactor: update DownloadFileFromUrlTransformation to extend ColumnsT…

635d4c4

…ransformationWithTarget and improve file download handling

dannymeijer added the enhancement New feature or request label Dec 12, 2024

dannymeijer self-assigned this Dec 12, 2024

dannymeijer marked this pull request as ready for review December 12, 2024 15:25

dannymeijer requested a review from a team as a code owner December 12, 2024 15:25

riccamini reviewed Dec 13, 2024

View reviewed changes

src/koheesio/steps/download_file.py Outdated Show resolved Hide resolved

src/koheesio/steps/download_file.py Outdated Show resolved Hide resolved

src/koheesio/steps/download_file.py Outdated Show resolved Hide resolved

dannymeijer added 3 commits December 13, 2024 18:02

refactor: enhance code readability by formatting multi-line arguments…

c3faf87

… and adding type hints in tests

chore: address review comments (mild refactoring of write_mode method…

f2101ae

…s) + added additional tests + documentation updates

chore: remove outdated file

c18cf66

dannymeijer requested a review from riccamini December 13, 2024 17:09