🌞🌞How to Quickly Identify and Add Newly Added Files to DVC?

[Leo5050xvjf]

Imagine working with a dataset containing one million entries (e.g., .bmp files), and you've just added 1,000 new files to it. Is there a way to swiftly incorporate these 1,000 new files into DVC? Or must I resort to using commands to iteratively search for all .bmp files and attempt to add them to DVC?

What strategies or commands can be used to efficiently manage the addition of new datasets in such a large-scale project using DVC?

Looking for insights or tips from those who have navigated similar challenges.

[skshetry]

@Leo5050xvjf, Please read Modifying large datasets. Let me. know if you have any questions.

But yes, you can track a single dataset and incorporate new files to it.

dvc add dataset # track a dataset

# to only update with new files in /dataset/new-files directory
dvc add dataset/new-files

The only downside is that you'd have to provide filenames or directories to update explicitly (dvc add dataset will go try to go through all your files).

EDIT: I forgot to mention that it can even work virtually, you don't have to download everything to even update or remove items from the tracked dataset.

[Leo5050xvjf]

Hello @skshetry,

Thank you for providing the tutorial link; I have completed reading it. I'd like to clarify my description further. Suppose I have added 500 new data files (.bmp) that might be distributed across different folders. I am looking for a way to quickly add these 500 data files similar to using a command like dvc add *.bmp, which would allow adding files such as /data/folder1/1.bmp, /data/folder2/2.bmp to DVC tracking.

Although using dvc data status --granular indeed helps to quickly understand the current status of the dataset, I'm curious if there is a specific command that can quickly add "modified status" data in a manner similar to dvc add. At the moment, it seems I can only achieve my goal by parsing the output from dvc data status --granular.

Leo

[skshetry]

dvc add data only adds modified data, so it already does that. It does not hash unchanged files, and does not transfer them to the cache. Unfortunately, it is slow (See #10337) at million file entries (YMMV).

What cache.type do you use? Are you using symlink, hardlink or using the default?

[Leo5050xvjf]

Hello,

Are you using symlink, hardlink or using the default? => default

Suppose I currently have a data directory containing an .iso file (2.4GB) like this:

data/
`-- ubuntu-18.04.6-desktop-amd64.iso

Then I initialize the entire project as follows:

1. git init
2. dvc init
3. dvc add data
4. git add .
5. git commit -m "ver1"

The dvc add data command takes about one minute to execute.

Next, I add a new file named a.txt to the data directory, resulting in the following structure:

data/
|-- a.txt
`-- ubuntu-18.04.6-desktop-amd64.iso

After that, when I enter:

dvc add data

It still takes about one minute to compute.

However, if I only execute dvc add data/a.txt, it completes very quickly.

My point is that DVC seems to lack a command for quickly adding a.txt. I can only expedite the addition by executing dvc add data/a.txt. Even though dvc data status --granular displays all the modifications, I can't quickly add those data. The execution time is still influenced by the 2.4GB file, right? If I've misunderstood, please let me know.

But this becomes highly inconvenient when I'm adding hundreds to thousands of files. With Git, even if the modified files are spread across various directories, I can still process them using git add . .However, it seems DVC does not offer a similar capability.

I want to confirm a question: does dvc add data recalculate the hash values for all files under the data directory once again?

[skshetry]

does dvc add data recalculate the hash values for all files under the data directory once again?

No, it does not re-hash again. If the file's mtime has not changed, it's not going to be rehashed.

Could you please share the cprofiling data from dvc add data?

You need to add --cprofile-dump=<filename> to the command.

dvc add data --cprofile-dump add.prof

This will create a add.prof file that you can attach here. You can also visualize that yourself using snakeviz.

pip install snakeviz
snakeviz add.prof

See https://github.com/iterative/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC#profiling-dvc.

[shcheklein]

hi @Leo5050xvjf , it might be a good scenario for the new upcoming DVCx release. If you are interested to discuss and potentially be one of the teams trying it first - please get back to me and let's schedule a chat :). It would be great to learn more about your use case and I hope I can give you more insights as well. Ping me at ivan [at] iterative.ai.