file: support exporting files as a symlink #819

skshetry · 2025-01-15T06:09:38Z

Closes #807.

Example Usage:

dc.export_files("output_dir", link_type="symlink")

cloudflare-workers-and-pages · 2025-01-15T06:10:41Z

Deploying datachain-documentation with Cloudflare Pages

Latest commit:	`2d9547f`
Status:	✅ Deploy successful!
Preview URL:	https://974724d7.datachain-documentation.pages.dev
Branch Preview URL:	https://symlink-export.datachain-documentation.pages.dev

View logs

skshetry · 2025-01-15T06:11:34Z

src/datachain/lib/file.py

+        if link_type == "symlink":
+            try:
+                return self._symlink_to(dst)
+            except OSError as exc:
+                if exc.errno not in (errno.ENOTSUP, errno.EXDEV, errno.ENOSYS):
+                    raise


What should we do in case of overwrite?

Looks like export allows you to overwrite files over and over again, but this _symlink_to will fail with FileExistsError.

skshetry · 2025-01-15T06:14:13Z

src/datachain/lib/file.py

+            self.ensure_cached()
+            source = self.get_local_path()
+            assert source, "File was not cached"
+        elif self.source.startswith("file://"):


Using self.source.startswith("file://") to avoid creating a filesystem.

codecov · 2025-01-15T06:25:57Z

Codecov Report

Attention: Patch coverage is 63.15789% with 7 lines in your changes missing coverage. Please review.

Project coverage is 87.47%. Comparing base (88737b6) to head (2d9547f).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/datachain/lib/file.py	61.11%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #819      +/-   ##
==========================================
- Coverage   87.54%   87.47%   -0.07%     
==========================================
  Files         128      128              
  Lines       11326    11344      +18     
  Branches     1533     1538       +5     
==========================================
+ Hits         9915     9923       +8     
- Misses       1026     1033       +7     
- Partials      385      388       +3

Flag	Coverage Δ
datachain	`87.41% <63.15%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

skshetry · 2025-01-15T07:59:46Z

tests/unit/lib/test_file.py

+    assert (tmp_path / "dir" / "myfile.txt").is_symlink()
+
+    dst = Path(file.get_local_path()) if use_cache else path
+    assert (tmp_path / "dir" / "myfile.txt").resolve() == dst


Using resolve() instead of readlink() because WindowsPath.readlink() returns an extended path that fails to compare (note the //?/ prefix).

-- WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw2/default_working_dir0/.datachain/cache/29/89fe8cd7eda0a1e2e28bf837e137f5bdd8d20c506348f67f5671f5a036d529') ++ WindowsPath('//?/C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/popen-gw2/default_working_dir0/.datachain/cache/29/89fe8cd7eda0a1e2e28bf837e137f5bdd8d20c506348f67f5671f5a036d529')

shcheklein · 2025-01-15T20:24:50Z

src/datachain/lib/dc.py

@@ -2418,6 +2419,7 @@ def export_files(
        signal="file",
        placement: FileExportPlacement = "fullpath",
        use_cache: bool = True,
+        link_type: Literal["copy", "symlink"] = "copy",


can we update / add docs while we do this (or as a followup)

done in 2d9547f

shcheklein

while we on this - any ideas about a better name for the export_file? + it would be great to update docs. Thanks for a quick turnaround @skshetry

skshetry · 2025-01-16T02:59:07Z

while we on this - any ideas about a better name for the export_file?

I could not think of a good name tbh. So, I left it at that.

Some of my suggestions are: download_files, checkout_files, get_files and link_files.
I prefer download_files makes more sense if copy is going to be a default link type, and link_files if link types other than copy is going to be a default link type.

skshetry · 2025-01-16T03:00:40Z

src/datachain/lib/dc.py

@@ -2418,6 +2419,7 @@ def export_files(
        signal="file",
        placement: FileExportPlacement = "fullpath",
        use_cache: bool = True,
+        link_type: Literal["copy", "symlink"] = "copy",


Another thing I forgot to mention - if files fail to symlink, it will always fall back to copy.

Closes #807.

skshetry temporarily deployed to internal January 15, 2025 06:09 — with GitHub Actions Inactive

skshetry requested a review from a team January 15, 2025 06:09

skshetry commented Jan 15, 2025

View reviewed changes

skshetry force-pushed the symlink-export branch from 3335d18 to 6e6aa3c Compare January 15, 2025 06:13

skshetry temporarily deployed to internal January 15, 2025 06:14 — with GitHub Actions Inactive

skshetry commented Jan 15, 2025

View reviewed changes

skshetry force-pushed the symlink-export branch from 6e6aa3c to 876d6aa Compare January 15, 2025 07:28

skshetry temporarily deployed to internal January 15, 2025 07:28 — with GitHub Actions Inactive

skshetry force-pushed the symlink-export branch from 876d6aa to 3136a3b Compare January 15, 2025 07:49

skshetry temporarily deployed to internal January 15, 2025 07:49 — with GitHub Actions Inactive

skshetry commented Jan 15, 2025

View reviewed changes

skshetry force-pushed the symlink-export branch from 3136a3b to 11925cb Compare January 15, 2025 08:28

skshetry temporarily deployed to internal January 15, 2025 08:28 — with GitHub Actions Inactive

shcheklein reviewed Jan 15, 2025

View reviewed changes

shcheklein approved these changes Jan 15, 2025

View reviewed changes

skshetry commented Jan 16, 2025

View reviewed changes

file: support exporting files as a symlink

2d9547f

Closes #807.

skshetry force-pushed the symlink-export branch from 11925cb to 2d9547f Compare January 16, 2025 04:01

skshetry temporarily deployed to internal January 16, 2025 04:01 — with GitHub Actions Inactive

skshetry merged commit aad99e2 into main Jan 16, 2025
36 of 38 checks passed

skshetry deleted the symlink-export branch January 16, 2025 04:17

dreadatour pushed a commit that referenced this pull request Jan 27, 2025

file: support exporting files as a symlink (#819)

d3b1619

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

file: support exporting files as a symlink #819

file: support exporting files as a symlink #819

skshetry commented Jan 15, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 15, 2025 •

edited

Loading

skshetry Jan 15, 2025

skshetry Jan 15, 2025 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading

skshetry Jan 15, 2025 •

edited

Loading

shcheklein Jan 15, 2025

skshetry Jan 16, 2025

shcheklein left a comment

skshetry commented Jan 16, 2025

skshetry Jan 16, 2025

file: support exporting files as a symlink #819

file: support exporting files as a symlink #819

Conversation

skshetry commented Jan 15, 2025 • edited Loading

Example Usage:

cloudflare-workers-and-pages bot commented Jan 15, 2025 • edited Loading

Deploying datachain-documentation with Cloudflare Pages

skshetry Jan 15, 2025

Choose a reason for hiding this comment

skshetry Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jan 15, 2025 • edited Loading

Codecov Report

skshetry Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

shcheklein Jan 15, 2025

Choose a reason for hiding this comment

skshetry Jan 16, 2025

Choose a reason for hiding this comment

shcheklein left a comment

Choose a reason for hiding this comment

skshetry commented Jan 16, 2025

skshetry Jan 16, 2025

Choose a reason for hiding this comment

skshetry commented Jan 15, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 15, 2025 •

edited

Loading

skshetry Jan 15, 2025 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading

skshetry Jan 15, 2025 •

edited

Loading