Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove legacy columns #263

Merged
merged 4 commits into from
Sep 11, 2024
Merged

Remove legacy columns #263

merged 4 commits into from
Sep 11, 2024

Conversation

rlamy
Copy link
Contributor

@rlamy rlamy commented Aug 8, 2024

This removes the following attributes or columns from various internal classes and tables: vtype, dir_type, owner_name, owner_id. Note that Node.dir_type is still used though.

The publicly visible, backwards-incompatible changes are:

  • File.vtype is removed
  • Existing listings table become invalid.

Part of #36

Copy link

cloudflare-workers-and-pages bot commented Aug 9, 2024

Deploying datachain-documentation with  Cloudflare Pages  Cloudflare Pages

Latest commit: fafe4b5
Status: ✅  Deploy successful!
Preview URL: https://2b2468dd.datachain-documentation.pages.dev
Branch Preview URL: https://rm-legacy-columns.datachain-documentation.pages.dev

View logs

@rlamy rlamy force-pushed the rm-legacy-columns branch from e61f4bf to 05a3585 Compare August 9, 2024 17:50
@rlamy rlamy force-pushed the rm-legacy-columns branch from 05a3585 to 7b9121e Compare August 27, 2024 16:29
Copy link

codecov bot commented Aug 27, 2024

Codecov Report

Attention: Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.

Project coverage is 87.09%. Comparing base (48a0de5) to head (fafe4b5).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/datachain/cli.py 25.00% 3 Missing ⚠️
src/datachain/listing.py 0.00% 0 Missing and 2 partials ⚠️
src/datachain/node.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #263      +/-   ##
==========================================
+ Coverage   87.07%   87.09%   +0.01%     
==========================================
  Files          92       92              
  Lines        9952     9925      -27     
  Branches     2037     2031       -6     
==========================================
- Hits         8666     8644      -22     
+ Misses        931      927       -4     
+ Partials      355      354       -1     
Flag Coverage Δ
datachain 87.04% <66.66%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rlamy rlamy force-pushed the rm-legacy-columns branch from 839b118 to c6a0471 Compare August 29, 2024 12:27
@rlamy rlamy marked this pull request as ready for review August 29, 2024 13:52
@rlamy rlamy mentioned this pull request Aug 30, 2024
@rlamy rlamy force-pushed the rm-legacy-columns branch from c6a0471 to ed47bef Compare August 30, 2024 19:10
Copy link
Member

@shcheklein shcheklein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Do we need to invalidate listing on the Studio side or remove them? Does it also invalidate datasets (do they have these file fields inherited from listings)?

@rlamy rlamy force-pushed the rm-legacy-columns branch from ed47bef to 4c30f90 Compare September 2, 2024 15:20
@rlamy
Copy link
Contributor Author

rlamy commented Sep 2, 2024

@shcheklein This makes existing listing tables invalid, but I think that's irrelevant after #294. For datasets, the only change is that File.vtype is removed, but it wasn't used before so I don't think it'll cause any issue. Old dataset tables will just have an extra "file__vtype" column that should be ignored everywhere (UI, queries, ...).

@shcheklein
Copy link
Member

@rlamy are we good to go here? :)

@rlamy rlamy force-pushed the rm-legacy-columns branch from 4c30f90 to d4e50ec Compare September 4, 2024 15:18
@rlamy
Copy link
Contributor Author

rlamy commented Sep 4, 2024

@shcheklein The datachain part works, but we need the Studio tests to pass...

@rlamy rlamy force-pushed the rm-legacy-columns branch 5 times, most recently from bb720ab to 8d15525 Compare September 10, 2024 16:04
@rlamy rlamy merged commit 1480757 into main Sep 11, 2024
37 of 38 checks passed
@rlamy rlamy deleted the rm-legacy-columns branch September 11, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants