Stable Diffusion Dataset

[amasin2111]

When we were trying to download the data set using the script laion400m-filtered-download-images.sh, we were getting an error that the source directory doesn't exist. Specifically, below command is failing,
"rclone copy mlc-training:mlcommons-training-wg-public/stable_diffusion/datasets/laion-400m/moments-webdataset-filtered/ ${OUTPUT_DIR} --include="*.tar" -P"

[amasin2111]

Hi, were you able to get around it

[morphine00]

@ahmadki both me and @nathanw-mlc tested the rclone commands and the scripts, and the data exists in the bucket (see attached)

What I did notice is that even the original scrips assumes that the destination directory /datasets/etcetc can be created, but unless the user is root, they won't have permissions to do so. Maybe this is the reason why it fails?

[amasin2111]

Hi,
I have used the same commands but still observing the same issue

[nathanw-mlc]

Can those having issues please share the result of rclone version.

[amasin2111]

Here is the version

[nathanw-mlc]

I just noticed that the update to the Dockerfile uses apt-get install to install Rclone. This install method installs an old version of Rclone (rclone v1.53.3-DEV) that doesn't process the rclone config create command correctly, resulting in Rclone attempting to connect to an AWS S3 bucket with the provided credentials, rather than a Cloudflare R2 bucket. Users need to be running v1.6x.x. To make that happen, the Dockerfile should install Rclone with the install command we provide for all Rclone instructions: sudo -v ; curl https://rclone.org/install.sh | sudo bash

[amasin2111]

It worked for me, other users might have to clean the config files, before retrying with new rclone version

[hiwotadese]

@ahmadki can we fix Dockerfile with @nathanw-mlc sugguestion sudo -v ; curl https://rclone.org/install.sh | sudo bash?

[ahmadki]

I genuinely dislike piping scripts from the internet into bash. Not only does it pose a security risk, but we also need to freeze rclone to a specific version.

#757 should work better.

[ShriyaPalsamudram]

Closing because #757 is merged.