Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heavy load on the gs://gresearch/robotics bucket #79

Open
Toradus opened this issue Jul 11, 2024 · 0 comments
Open

Heavy load on the gs://gresearch/robotics bucket #79

Toradus opened this issue Jul 11, 2024 · 0 comments

Comments

@Toradus
Copy link

Toradus commented Jul 11, 2024

First thanks for the awesome work of unifying the datasets!

Im currently trying to train a model using datasets from the gs://gresearch/robotics repo, which worked very good for a long time but since the last few days everything kind of slowed down.
Is there currently a heavy load on the google Server or is this a problem on my end?
I tried 2 different Clusters and both showed bad results for tfds.benchmark:

import tensorflow as tf
import tensorflow_datasets as tfds

tf.config.set_visible_devices([], "GPU")
builder = tfds.builder_from_directory(builder_dir="gs://gresearch/robotics/droid/1.0.0")
ds = builder.as_dataset(split="train[:95%]")
ds = ds.shuffle(1000) # not needed but important for training so i added it
ds = ds.batch(32).prefetch(buffer_size=tf.data.AUTOTUNE)
tfds.benchmark(ds, batch_size=256)

it = iter(ds)
for i in range(2):
    episode = next(it)

# steps = list(episode['steps'])
print("done")
_________________________

Examples/sec (First included) 182.94 ex/sec (total: 701440 ex, 3834.31 sec)
Examples/sec (First only) 2.71 ex/sec (total: 256 ex, 94.32 sec)
Examples/sec (First excluded) 187.48 ex/sec (total: 701184 ex, 3739.99 sec)

And also bad results for "Computing dataset statistics" using the Octo Dataloader for pytorch (from ~200it/s some weeks ago to ~1.66s/it)

Just want to make sure if it is a problem on my end (e.g. change in TF versions / Net connection) or if currently many people load data and it is only heavy load.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant