Handle permission error properly when checking for file #856

amritghimire · 2025-01-24T03:20:19Z

main > #856 (this) > #857

Currently, we had blanket catch for exception when trying to check the
file using _isfile. As a result, the exception stacktrace was repeated
and catching the exception in script was difficult as we had to capture
different exception. This convert the error to datachain native error
that can be captured safely and proceed accordingly.

This is first step toward handling #600

Currently, we had blanket catch for exception when trying to check the file using _isfile. As a result, the exception stacktrace was repeated and catching the exception in script was difficult as we had to capture different exception. This convert the error to datachain native error that can be captured safely and proceed accordingly. This is first step toward handling #600

cloudflare-workers-and-pages · 2025-01-24T03:20:21Z

Deploying datachain-documentation with Cloudflare Pages

Latest commit:	`af393cd`
Status:	✅ Deploy successful!
Preview URL:	https://6ea56970.datachain-documentation.pages.dev
Branch Preview URL:	https://amrit-handle-permission-erro.datachain-documentation.pages.dev

View logs

shcheklein · 2025-01-24T04:04:29Z

Currently, we had blanket catch for exception when trying to check the
file using _isfile. As a result, the exception stacktrace was repeated
and catching the exception in script was difficult as we had to capture
different exception

I'm not sure I understand this ... could you give an example please?

amritghimire · 2025-01-24T04:58:09Z

Currently, we had blanket catch for exception when trying to check the
file using _isfile. As a result, the exception stacktrace was repeated
and catching the exception in script was difficult as we had to capture
different exception

I'm not sure I understand this ... could you give an example please?

https://github.com/iterative/datachain/pull/856/files#diff-ba026f8084c104850332b7e62ae0741b251f65b67d0271ce5870f4794a1346bfL102

shcheklein · 2025-01-24T05:00:45Z

https://github.com/iterative/datachain/pull/856/files#diff-ba026f8084c104850332b7e62ae0741b251f65b67d0271ce5870f4794a1346bfL102

is it an example?

amritghimire · 2025-01-24T05:13:45Z

#856 (files)

is it an example?

Yes, as you see we had

except:

with supressed lint. That means no matter what the error we were returning false. So, if we did not have the permission, the script proceeded.

codecov · 2025-01-24T11:18:59Z

Codecov Report

Attention: Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.61%. Comparing base (13e5c13) to head (af393cd).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/datachain/lib/listing.py	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #856      +/-   ##
==========================================
+ Coverage   87.53%   87.61%   +0.07%     
==========================================
  Files         128      128              
  Lines       11369    11382      +13     
  Branches     1538     1540       +2     
==========================================
+ Hits         9952     9972      +20     
+ Misses       1029     1023       -6     
+ Partials      388      387       -1

Flag	Coverage Δ
datachain	`87.54% <92.85%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

amritghimire · 2025-01-24T11:45:50Z

https://github.com/iterative/datachain/pull/767/files#diff-ba026f8084c104850332b7e62ae0741b251f65b67d0271ce5870f4794a1346bfR102-R103

This code looks problematic but uncaught because we just ignored everything with except. The _isfile doesn't work in windows for glob as a result as seen in https://github.com/iterative/datachain/actions/runs/12948522550/job/36117365039?pr=856

dreadatour · 2025-01-25T15:58:27Z

src/datachain/lib/listing.py

+    except FileNotFoundError:
        return False
+    except REMOTE_ERRORS as e:
+        raise ClientError(
+            message=str(e),
+            error_code=getattr(e, "code", None),
+        ) from e


Do we want to catch other exceptions here? 🤔

I think that should be the burden of outer call or calling function to capture exception that are totally unexpected.

shcheklein · 2025-01-25T21:41:07Z

This code looks problematic but uncaught because we just ignored everything with except. The _isfile doesn't work in windows for glob as a result as seen in https://github.com/iterative/datachain/actions/runs/12948522550/job/36117365039?pr=856

yep, but that's why I guess we were just ignoring a wide range of exceptions and the rest of the code is built to handle that path as a non-file

should we just try to catch Remote exceptions that are relevant to credentials, etc. And then do a wide open except:? wdyt?

skshetry · 2025-01-26T04:13:42Z

src/datachain/lib/listing.py

@@ -90,6 +91,10 @@ def _isfile(client: "Client", path: str) -> bool:
    Returns True if uri points to a file


@shcheklein, was _isfile() added just for making it work with Google Cloud?

If so, I'd recommend extending GCSFileSystem._is_file (my preference), or add GCSClient._is_file.

Whitelisting exceptions on a fs-agnostic/generic code is not a good thing to do.
get_listing could raise any unknown exceptions that's not FileNotFoundError in _is_file as a ClientError.

no, I think it is a general code (there are special zero-byte files across all clouds I think) - we need to filter them

Btw, we stopped doing these checks in dvc. We ignore files that end with slashes.

yes, so that's the point of this call - to always treat it as a directory AFAIR

I guess in some cases (not sure if this works the same across all clients) it returns a file/ even if ask it to check file if zero-byte object exists

I think this warrants separate discussion if we want to include _isfile or not. Merging the PR for now.

The discussion is not limited to only _isfile.
I am arguing that _is_file should not raise ClientError, or handle certain set of exceptions. _is_file should raise any exception that is not FileNotFoundError, and get_listing should convert it to ClientError without any filtering. There's no need for whitelisting.

_isfile should be very close to fs._is_file.

I guess in this case we can consider is_file more like client.is_file that already abstracts out fs - thus it is probably fine to convert exceptions if that is relevant

shcheklein · 2025-01-26T04:16:49Z

src/datachain/lib/listing.py

@@ -90,6 +91,10 @@ def _isfile(client: "Client", path: str) -> bool:
    Returns True if uri points to a file
    """
    try:
+        if "://" in path:


why is this needed, btw?

This is because when the s3 or other schema was in uppercase, the process would fail since it didn't match the specified regex. Its just that we swallowed the exception in test cases previously.

Could you point me to those tests that use uppercase scheme?

datachain/tests/func/test_ls.py

Line 78 in 60aed77

def test_ls_sources_scheme_uppercased(cloud_test_catalog, cloud_type, capsys):

this still very weird place to do this kind of transformation ... how was it handled by ls downstream? Okay, it's swallowed here means we are losing some optimization - by ls still worked right? where do we handle this usually?

amritghimire · 2025-01-27T03:04:38Z

should we just try to catch Remote exceptions that are relevant to credentials, etc. And then do a wide open except:? wdyt?

A wide open except: would swallow a lot of issues IMO.

* Handle permission error properly when checking for file Currently, we had blanket catch for exception when trying to check the file using _isfile. As a result, the exception stacktrace was repeated and catching the exception in script was difficult as we had to capture different exception. This convert the error to datachain native error that can be captured safely and proceed accordingly. This is first step toward handling #600 * Convert scheme to lower * Handle case for glob in windows

amritghimire self-assigned this Jan 24, 2025

amritghimire requested review from ilongin, dreadatour and a team January 24, 2025 03:20

Merge branch 'main' into amrit/handle-permission-error-properly

66c438c

amritghimire mentioned this pull request Jan 24, 2025

Filter logging to hide massive stacktrace for access issue #857

Merged

Merge branch 'main' into amrit/handle-permission-error-properly

9c3afef

Convert scheme to lower

911d360

amritghimire requested a review from shcheklein January 25, 2025 01:35

dreadatour approved these changes Jan 25, 2025

View reviewed changes

skshetry reviewed Jan 26, 2025

View reviewed changes

shcheklein reviewed Jan 26, 2025

View reviewed changes

Handle case for glob in windows

af393cd

amritghimire merged commit b1d4f2a into main Jan 27, 2025
37 checks passed

amritghimire deleted the amrit/handle-permission-error-properly branch January 27, 2025 06:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle permission error properly when checking for file #856

Handle permission error properly when checking for file #856

amritghimire commented Jan 24, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 24, 2025 •

edited

Loading

shcheklein commented Jan 24, 2025

amritghimire commented Jan 24, 2025

shcheklein commented Jan 24, 2025

amritghimire commented Jan 24, 2025

codecov bot commented Jan 24, 2025 •

edited

Loading

amritghimire commented Jan 24, 2025

dreadatour Jan 25, 2025

amritghimire Jan 27, 2025

shcheklein commented Jan 25, 2025

skshetry Jan 26, 2025 •

edited

Loading

shcheklein Jan 26, 2025

skshetry Jan 26, 2025

shcheklein Jan 26, 2025

shcheklein Jan 26, 2025

amritghimire Jan 27, 2025

skshetry Jan 27, 2025

skshetry Jan 27, 2025

shcheklein Jan 28, 2025

shcheklein Jan 26, 2025

amritghimire Jan 27, 2025

skshetry Jan 27, 2025

amritghimire Jan 27, 2025

shcheklein Jan 28, 2025

amritghimire commented Jan 27, 2025

		@@ -90,6 +91,10 @@ def _isfile(client: "Client", path: str) -> bool:
		Returns True if uri points to a file

Handle permission error properly when checking for file #856

Handle permission error properly when checking for file #856

Conversation

amritghimire commented Jan 24, 2025 • edited Loading

main > #856 (this) > #857

cloudflare-workers-and-pages bot commented Jan 24, 2025 • edited Loading

Deploying datachain-documentation with Cloudflare Pages

shcheklein commented Jan 24, 2025

amritghimire commented Jan 24, 2025

shcheklein commented Jan 24, 2025

amritghimire commented Jan 24, 2025

codecov bot commented Jan 24, 2025 • edited Loading

Codecov Report

amritghimire commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shcheklein commented Jan 25, 2025

skshetry Jan 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amritghimire commented Jan 27, 2025

amritghimire commented Jan 24, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 24, 2025 •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading

skshetry Jan 26, 2025 •

edited

Loading