Refactor DataChain.from_storage() to use new listing generator

[ilongin]

Refactoring DataChain.from_storage() to use new listing generator instead of calling underlying DatasetQuery indexing step which uses deprecated listing codebase.

For each listing requested in .from_storage() method, new dataset will be created with lst_ prefix and uri, e.g
DataChain.from_storage("s3://ldb-public/dogs-cats") -> this will create dataset with name lst_s3://ldb-public/dogs-cats which holds listing data.

Schema of listing dataset will always be {"file": "File@v1"}, but user can specify specific type (e,g ImageFile or TextFile) by adding argument type e.g .from_storage("s3://ldb-public", type="text") which would return chain with schema consisting of that specific type instead of generic File. This way we can, for example, filter out images from chain and set correct image type, e.g

# listing is done in first from_storage() call where schema is File but reading it as ImageFile
img_dc = DataChain.from_storage("gs://datachain-demo", type="image").filter(C("file.path").glob("*.jpg"))
# using cached listing from above but reading it as TextFile this time
txt_dc = DataChain.from_storage("gs://datachain-demo", type="text").filter(C("file.path").glob("*.txt"))  

Partials listing is also done without partials table which makes them obsolete.
Storage table is also obsolete as now all information is in that special listing dataset from above.

Next steps:

• Make listing caching lazy
• Refactor CLI to use DataChain.from_storage() to enlist sources
• Remove obsolete codebase

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	62045cd
Status:	✅  Deploy successful!
Preview URL:	https://d42b5923.datachain-documentation.pages.dev
Branch Preview URL:	https://ilongin-266-refactor-from-st.datachain-documentation.pages.dev

View logs

[codecov]

Codecov Report

Attention: Patch coverage is 93.67089% with 5 lines in your changes missing coverage. Please review.

Project coverage is 86.79%. Comparing base (315816e) to head (62045cd).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/datachain/lib/meta_formats.py	0.00%	3 Missing ⚠️
src/datachain/lib/dc.py	91.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #294      +/-   ##
==========================================
+ Coverage   86.74%   86.79%   +0.04%     
==========================================
  Files          92       92              
  Lines       10072    10124      +52     
  Branches     2046     2055       +9     
==========================================
+ Hits         8737     8787      +50     
- Misses        986      988       +2     
  Partials      349      349              

Flag	Coverage Δ
datachain	86.72% <93.67%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shcheklein]

It seems just a few things left. I'm fine if we hide client_config / catalog under kwargs for now (as it was before). session in some APIs is fine as well.

Re the globs - if it was the same logic as before - only /* is supported - I'm fine with that for this PR

There are a few questions left by Ronan and I had a new minor one I think.

I resolved almost everything else.

Let's get this merged soon :) Thanks for your patience @ilongin !

[dmpetrov]

Looks great! PLease remove the kwargs and config and it;s good to merge.