-
Notifications
You must be signed in to change notification settings - Fork 28
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
cc40c4c
commit 55b24d3
Showing
3 changed files
with
146 additions
and
155 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
from pydantic import DirectoryPath | ||
|
||
from koheesio.spark.transformations import ColumnsTransformationWithTarget | ||
|
||
|
||
class DownloadFileFromUrlTransformation(ColumnsTransformationWithTarget): | ||
""" | ||
Downloads content from URLs in the specified column and stores the downloaded file paths in a new column. | ||
Parameters | ||
---------- | ||
columns : ListOfColumns | ||
The column (or list of columns) containing the URLs to download. | ||
target_column : Optional[str], optional, default=None | ||
The name of the column to store the downloaded file paths. If not provided, the result will be stored in the source column. | ||
download_path : DirectoryPath | ||
The local directory path where the files will be downloaded. | ||
""" | ||
|
||
download_path: DirectoryPath | ||
|
||
def func(self, column): | ||
pass |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters