Skip to content

NSFS on Kubernetes

Jump to bottom
shirady edited this page Jun 13, 2023 · 33 revisions

NSFS (short for Namespace-Filesystem) is a capability to use a shared filesystem (mounted in the endpoints) for the storage of S3 buckets, while keeping a 1-1 mapping between Object and File.

Step 1 - Deploy in Kubernetes

This feature is currently under development, and it is recommended to use latest weekly master releases, starting from the build tagged with master-20210419.

Refer to this guide: https://github.com/noobaa/noobaa-core/wiki/Weekly-Master-Builds

Step 2 - Create PVC

For NSFS to work it requires a PVC for the filesystem, with a ReadWriteMany accessMode so that we can scale the endpoints to any node in the cluster and still be able to share the volume.

Ideally this PVC will be allocated by a provisioner, such as the CSI provisioner of rook-ceph.cephfs.csi.ceph.com (link).

If you don't have a CSI provisioner you can just set up a local volume manually using this guide: https://github.com/noobaa/noobaa-core/wiki/NSFS-using-a-Local-PV-on-k8s

Step 3 - Access Permissions

S3 access will be determined by the mapping of each S3 account to UID/GID (see Step 7 - Create Account) and the access of that UID/GID to the directories and files in the filesystem. The filesystem admin should set up the ACLs/unix-permissions for the mounted FS path to the needed UIDs/GIDs that would be used to access it.

For dev/test the simplest way to set this up is to give full access to all:

mkdir -p /nsfs/bucket-path
chmod -R 777 /nsfs/bucket-path

NOTE: on minikube run minikube ssh and then run the above with sudo

# on minikube
minikube ssh "mkdir -p /nsfs/bucket-path"
minikube ssh "chmod -R 777 /nsfs/bucket-path"

Step 4 - Create NSFS Resource

A namespace resource is a configuration entity that represents the mounted filesystem in the noobaa system.

You need to provide it with some information:

  • namespace-store-name - Choose how to name it, perhaps follow the same name as the PVC or the Filesystem. You will use this name later when creating buckets that use this filesystem.
  • pvc-name - The name of the pvc in which the filesystem resides.
  • fs-backend (optional) - When empty will assume basic POSIX filesystem only. Supported backend types: NFSv4, CEPH_FS, GPFS. Setting the more specific backend will allow optimization based on the capabilities of the underlying filesystem.

Here is an example of calling this API:

noobaa namespacestore create nsfs fs1 --pvc-name='nsfs-vol' --fs-backend='GPFS'

NOTE: on minkube do not use the fs-backend flag, leave it empty.

# on minikube
noobaa namespacestore create nsfs fs1 --pvc-name='nsfs-vol'

Step 5 - Create Bucket(s) - via noobaa API

NSFS Buckets are like creating an "export" for a filesystem directory in the S3 service.

The following API call will create a bucket with the specified name, and redirect it to a specified path from the NSFS resource that was created in Step 4 - Create NSFS Resource.

noobaa api bucket_api create_bucket '{
  "name": "fs1-jenia-bucket",
  "namespace":{
    "write_resource": { "resource": "fs1", "path": "bucket-path/" },
    "read_resources": [ { "resource": "fs1", "path": "bucket-path/" }]
  }
}'

Step 6 - Add bucket policy

  • We should update the bucket policy using admin account. Use the admin credentials from noobaa status --show-secrets
  • endpoint-url is noobaa endpoint address it can be taken from NodesPorts address from noobaa status or an address in localhost after port-forward (for example if you're using kubectl port-forward -n default service/s3 12443:443 it will be --endpoint-url=https://localhost:12443.
aws --endpoint-url=<insert_addresss> --no-verify-ssl s3api put-bucket-policy --bucket fs1-jenia-bucket --policy file://policy.json

policy.json is a JSON file in the current directory that specifies the object to delete:

{
"Version":"2012-10-17",
"Statement":[
        {
        "Sid":"id-1",
        "Effect":"Allow",
        "Principal":"*",
        "Action":["s3:*"],
        "Resource":["arn:aws:s3:::*"]
        }
]
}

Step 7 - Create Account(s)

Create accounts with NSFS configuration:

  • Map the account to a UID/GID
  • Set up the directory for new buckets created from S3 for this account (TBD)
  • default_resource should be the same as NSFS resource (see Step 4 - Create NSFS Resource).
  • new_buckets_path is the path in which new buckets created via s3 will be placed.
  • nsfs_only is a boolean field that defines the access permissions of an account to non-NSFS buckets.

Here is an example:

noobaa api account_api create_account '{
    "email": "[email protected]",
    "name": "jenia",
    "has_login": false,
    "s3_access": true,
    "default_resource": "fs1",
    "nsfs_account_config": {
      "uid": 1001,
      "gid": 0,
      "new_buckets_path": "/",
      "nsfs_only": false
     }
  }'

Create account returns a response with S3 credentials:

INFO[0001] ✅ RPC: account.create_account() Response OK: took 205.7ms 
access_keys:
- access_key: *NOOBAA_ACCOUNT_ACCESS_KEY*
  secret_key: *NOOBAA_ACCOUNT_SECRET_KEY*

You can also perform a list accounts command in order to see the configured NSFS accounts (besides all other accounts of the system)

noobaa api account_api list_accounts {}

If you are interested in a particular account you can read its information directly by email:

noobaa api account_api read_account '{"email":"[email protected]"}'

Step 8 - Connect and configure S3 Client

Configure the S3 client application and access the FS via S3 from the endpoint. Use the S3 credentials, access_key and secret_key, resulted from step 7 of "Create Account(s)".

Application S3 config:

AWS_ACCESS_KEY_ID=*NOOBAA_ACCOUNT_ACCESS_KEY*
AWS_SECRET_ACCESS_KEY=*NOOBAA_ACCOUNT_SECRET_KEY*
S3_ENDPOINT=s3.noobaa.svc (or nodePort address from noobaa status or address after kubectl port forwarding see comment above)
BUCKET_NAME=fs1-jenia-bucket

As we can create different accounts, it will be helpful to have these keys and end-points configured as an alias which can be used in step 9. For Example:

alias s3-user-1='AWS_ACCESS_KEY_ID=NsFsisNamEFlSytm AWS_SECRET_ACCESS_KEY=HiN00baa0nK8SDfsyV+VLoGK6ZMyCEDvklQCqW0 aws --endpoint "NodePort address"  --no-verify-ssl s3'

Step 9 - Create Bucket(s) - via S3

Use the S3 client configured in step 8 to create new buckets under the new_buckets_path in the default_resource configured by the requesting account.

S3 CLI tool is the part of alias created in step 8.

s3-user-1 mb s3://test-bucket

A new filesystem directory called "test-bucket" will be created by noobaa.

Based on the input we provided in this guide, "test-bucket" directory can be seen in /nsfs/fs1 of "endpoint" pod.

Clone this wiki locally