From 439b3feda8b14c268717e2138cb760fd207dd312 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 11:23:05 +0200 Subject: [PATCH 1/9] Clarify the use of `--no-copy` flag in out docs --- .../2_Using Datastores/2_creating-datastores.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index d6bda760..fe402ace 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -113,15 +113,10 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin Example: `grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy` -In certain cases, your S3 bucket may fit one (or both) of the following criteria: - -(1) the bucket is continually updating with new data which you want included in a Grid Datastore -(2) the bucket is particularly large (leading to long Datastore creation times) - -In these cases, you can pass the `--no-copy` flag to the `grid datastore create` command. This flag will -prevent Grid from making a copy of the dataset, which significantly speeds up Datastore creation time. +Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speeds up Datastore creation time, It will prevent grid from making a copy of the dataset. We recommend that you use this option when the your dataset size is greater than `500 GB`. :::info +We recommend that you use this flag when creating a private Datastore within a BYOC cluster. If your Datastore is created from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. When using this flag, you cannot remove files from your bucket. If you'd like to add files, please create a new version of the Datastore after you've added files to your bucket. From bae12373b5841e3d45036076010a6f145ed8eeb5 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 11:24:18 +0200 Subject: [PATCH 2/9] typo --- .../datastores/2_Using Datastores/2_creating-datastores.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index fe402ace..2b9f71b9 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -113,7 +113,7 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin Example: `grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy` -Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speeds up Datastore creation time, It will prevent grid from making a copy of the dataset. We recommend that you use this option when the your dataset size is greater than `500 GB`. +Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. We recommend that you use this option when the your dataset size is greater than `500 GB`. :::info We recommend that you use this flag when creating a private Datastore within a BYOC cluster. If your Datastore is created from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. From 663e36ff4a1e8e1681d56deeac98246a2118fe93 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 11:29:00 +0200 Subject: [PATCH 3/9] clarify private s3 --- .../2_Using Datastores/2_creating-datastores.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index 2b9f71b9..656712c2 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -113,10 +113,14 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin Example: `grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy` -Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. We recommend that you use this option when the your dataset size is greater than `500 GB`. +Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. We :::info -We recommend that you use this flag when creating a private Datastore within a BYOC cluster. If your Datastore is created from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. +We recommend that you use this flag when creating a private S3 Datastore within a BYOC cluster. + +Also consider using this option when the your dataset size is greater than `500 GB`. + +If your Datastore was from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. When using this flag, you cannot remove files from your bucket. If you'd like to add files, please create a new version of the Datastore after you've added files to your bucket. From 2c347063f3db5a15f237a90d11d10b0472cd6532 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 11:29:31 +0200 Subject: [PATCH 4/9] typo --- .../datastores/2_Using Datastores/2_creating-datastores.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index 656712c2..efa2f519 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -113,7 +113,7 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin Example: `grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy` -Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. We +Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. :::info We recommend that you use this flag when creating a private S3 Datastore within a BYOC cluster. From 6358a078e7556519aa049e5c34ba44cc419510d5 Mon Sep 17 00:00:00 2001 From: Femi <47154698+oojo12@users.noreply.github.com> Date: Mon, 18 Jul 2022 10:14:40 -0400 Subject: [PATCH 5/9] typo fix + multigpu note --- docs/getting-started/typical-workflow-cli-user.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/getting-started/typical-workflow-cli-user.md b/docs/getting-started/typical-workflow-cli-user.md index 32f1cb92..9bd1dfb9 100644 --- a/docs/getting-started/typical-workflow-cli-user.md +++ b/docs/getting-started/typical-workflow-cli-user.md @@ -325,7 +325,7 @@ cd grid-tutorials/getting-started pip install -r requirements.txt ``` -now run the following command to train a resnet18 on 2 GPUs +now run the following command to train a resnet18 on 1 GPU ```bash python flash-image-classifier.py \ @@ -387,6 +387,11 @@ Make sure you are in the /grid-tutorials/getting-started directory for the tutor Now kick off the run with grid run +:::note +To convert the below run into a multi-gpu Run please set the `--instance_type` parameter to a machine with multiple gpus +and update the `--gpus` argument to the desired number of GPUs. See [here](../platform/4_machines.md) for a list of machine types and their hardware specs. +::: + ```bash grid run --dependency_file ./requirements.txt \ --name cifar-tut-hpo \ From 6910ec4324fa2e887426b697fd6e0635fe3344d0 Mon Sep 17 00:00:00 2001 From: Femi <47154698+oojo12@users.noreply.github.com> Date: Mon, 18 Jul 2022 10:17:01 -0400 Subject: [PATCH 6/9] Update typical-workflow-cli-user.md --- docs/getting-started/typical-workflow-cli-user.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/typical-workflow-cli-user.md b/docs/getting-started/typical-workflow-cli-user.md index 9bd1dfb9..8b306126 100644 --- a/docs/getting-started/typical-workflow-cli-user.md +++ b/docs/getting-started/typical-workflow-cli-user.md @@ -389,7 +389,7 @@ Now kick off the run with grid run :::note To convert the below run into a multi-gpu Run please set the `--instance_type` parameter to a machine with multiple gpus -and update the `--gpus` argument to the desired number of GPUs. See [here](../platform/4_machines.md) for a list of machine types and their hardware specs. +and update the `--gpus` argument to the desired number of GPUs. See [here](../platform/4_machines.md) for a table of machine types and their hardware specs. ::: ```bash From e9f15a9882a5be145e7fe164c858bf11f68003d9 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 17:59:54 +0200 Subject: [PATCH 7/9] Improve the steps to create a private S3 bucket simpler Based on discussion with @panos-is ``` a. Naviagte to AWS WebUI -> IAM -> [Roles](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles) b. Create Role -> Custom trust policy c. Paste the output of the CLI into the `json` editor and click Next [Image STS] d. In the `Add permissions` page click Next e. The role name must begin with `grid-s3-access-` and click on Create role [Image Role] f. You will be redirected to IAM -> Roles. Select your role g. Click on Add Permissions -> Create Inline Policy and paste the permission policy from the CLI output after replacing the bucket name and click on Review Policy h. Enter an appropriate name, for example, my-permission-policy-for-grid. [Image Permission Policy] ``` --- docs/platform/3_credentials.md | 66 ++++++++++------------------------ 1 file changed, 19 insertions(+), 47 deletions(-) diff --git a/docs/platform/3_credentials.md b/docs/platform/3_credentials.md index c092433d..b10aca9a 100644 --- a/docs/platform/3_credentials.md +++ b/docs/platform/3_credentials.md @@ -135,65 +135,37 @@ When complete, please enter the role ARN: The `grid credential create` command will output the required trust and permission policies. - a. Leave the command prompt open while visiting the [AWS IAM Role - Console](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles) + a. Naviagte to AWS WebUI -> IAM -> [Role](https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-1#/roles) - b. Click "Create Role" + b. Create Role -> Custom trust policy ![](../../static/images/credentials/role-console.png) - c. Select the "custom trust policy" setting, and copy/paste the trust policy output - from the CLI into the `json` editor: + c. Paste the output of the `trust policy` from the CLI into the `json` editor and click Next ![](../../static/images/credentials/select_trusted_entity.png) - d. When prompted with the "add permissions to role" tab, click "Create Policy" to create a new permission - policy + d. In the `Add permissions` page click Next + + e. The role name must begin with `grid-s3-access-` and click on Create role + + ![](../../static/images/credentials/role_naming.png) + + f. You will be redirected to IAM -> Roles. Select your just created + + g. Click on Add Permissions -> Create Inline Policy. Paste the output `permission policy` from the CLI output after replacing your bucket name and click on Review Policy + + ![](../../static/images/credentials/permission_policy_naming.png) - ![](../../static/images/credentials/add_permissions_to_role.png) - - e. A new tab will open. Select the `json` tab, and - copy/paste the permission policy into the editor. Also replace the text - `` with the name of the private bucket to provide access to - (i.e. `gridai-demo-bucket`). - - ![](../../static/images/credentials/aws_permission_policy.png) - - f. Click "next" until reaching the "permission policy naming" tab. Enter an appropriate - name, for example: `my-permission-policy-for-grid`. - - ![](../../static/images/credentials/permission_policy_naming.png) - - g. Click "create policy". - - ![](../../static/images/credentials/permission_policy_created.png) - - h. Go back to the previous tab for attaching a permission policy to the role - - i. Refresh the list of policies by clicking the refresh symbol + h. Enter an appropriate name, for example: my-permission-policy-for-grid - j. Select the `my-permission-policy-for-grid` policy name and click "next". - -![](../../static/images/credentials/attach_permission_policy.png) - -### 3. Create AWS Role Name - -I then navigate through the prompts until I am presented with the "role naming" screen. At -this point, I create a role name **keeping in mind that the role must begin with the -prefix: `grid-s3-access-`**. In this case I name the role -`grid-s3-access-for-my-demo-bucket` and click create. - -![](../../static/images/credentials/role_naming.png) - - -I am then presented with a success screen! The role has been created. + ![](../../static/images/credentials/add_permissions_to_role.png) + ### 4. Input role ARN in Grid -![](../../static/images/credentials/role_created.png) - -I select the role and click on the icon next to the "role ARN" in order to -copy the value to my clipboard: +Within Roles -> Search for the role you created that began with `grid-s3-access-` and click on it +Click on the icon next to the "role ARN" in order to copy the value to my clipboard: ![](../../static/images/credentials/arn_copy.png) From 31350782bb0f7c860f2c1e21c35a5e2438d55ec3 Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Mon, 18 Jul 2022 18:07:35 +0200 Subject: [PATCH 8/9] femi suggestions --- .../datastores/2_Using Datastores/2_creating-datastores.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index efa2f519..61ce4f71 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -113,7 +113,7 @@ grid datastore create s3://ryft-public-sample-data/esRedditJson/ --name lightnin Example: `grid datastore create S3://ruff-public-sample-data/esRedditJson --no-copy` -Use `--no-copy` when you want to avoid creating dupicate copy of this data in your cluster account. Using this flag can significantly speed up Datastore creation time, It will prevent grid from making a copy of the dataset. +Use --no-copy when you want to avoid creating a duplicate copy of the data in your cluster account. Using this flag can significantly speed up Datastore creation time by preventing the copy process. :::info We recommend that you use this flag when creating a private S3 Datastore within a BYOC cluster. From 861f82409d9334d02426dcc26b0d28fad953b84f Mon Sep 17 00:00:00 2001 From: Krishna Kalyan Date: Tue, 19 Jul 2022 07:50:54 +0200 Subject: [PATCH 9/9] roberts comments --- .../datastores/2_Using Datastores/2_creating-datastores.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md index 61ce4f71..00424bed 100644 --- a/docs/features/datastores/2_Using Datastores/2_creating-datastores.md +++ b/docs/features/datastores/2_Using Datastores/2_creating-datastores.md @@ -120,7 +120,7 @@ We recommend that you use this flag when creating a private S3 Datastore within Also consider using this option when the your dataset size is greater than `500 GB`. -If your Datastore was from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. +If your Datastore was from a public S3 bucket, having a copy in the cluster can be advantageous if someone deletes or modifys the public bucket. `--no-copy` does not prevent you from deleting the file in S3 as thats managed by the user. If thats done the Datastore will show that the file still exists since the metadata is cached, On access S3 will indicate that the file does not exist anymore. When using this flag, you cannot remove files from your bucket. If you'd like to add files, please create a new version of the Datastore after you've added files to your bucket.