Skip to content
This repository has been archived by the owner on Apr 11, 2024. It is now read-only.

Fix the File to File transfer with source or destination is a pattern #26

Open
sunank200 opened this issue Mar 27, 2023 · 0 comments
Open

Comments

@sunank200
Copy link
Collaborator

sunank200 commented Mar 27, 2023

Describe the bug
There are two scenarios that need the fix:

  • When the source dataset is a file pattern or folder (eg: s3://dummy-bucket/folder/) and the destination dataset is a file, it should error out but currently, it would concatenate the file with that name.
  • While checking for file patterns we currently use pathlib library but we should also add checks using underlying provider library to correctly validate if it is a file or a file pattern.

Expected behavior
A clear and concise description of what you expected to happen.

  • Error out the source dataset is a file pattern or folder (eg: s3://dummy-bucket/folder/) and the destination dataset is a file.
    Eg:
transfer_file = UniversalTransferOperator(
        task_id="transfer_file",
        source_dataset=File(path=f"{s3_bucket}/uto/", conn_id="aws_default"),
        destination_dataset=File(path=f"{s3_bucket}/uto-dummy/sample-file.csv", conn_id="aws_default"),
    )
  • Check for file patterns using the underlying provider library to correctly validate if it is a file or a file pattern.

More details at: notion doc

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants