Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[STORY] Investigate S3 storage solutions #309

Closed
2 tasks
tclose opened this issue Jun 24, 2024 · 2 comments
Closed
2 tasks

[STORY] Investigate S3 storage solutions #309

tclose opened this issue Jun 24, 2024 · 2 comments
Assignees
@fxusyd
Labels
data-ingestion Ingestion of data from edge devices to XNAT pipelines story a unit of work

Comments

@tclose
Copy link
Contributor

tclose commented Jun 24, 2024 •
edited
Loading

Metadata

Epic: #271
Feature:
Feature Release:
Required knowledge: shallow or mid-level or deep

Description

Need to find a cheaper, reasonably performant storage solution for cloud object storage (e.g. S3 file-gateway, Objective FS)

Acceptance Criteria

  • 1. Settle on a solution that will be able to store all USyd's data requirements for next couple of years (e.g. TBP)
  • 2. Layout on transition plan as large-scale university storage moves to local cloud

Blocked by

  • [ ]
@github-actions github-actions bot added pipelines story a unit of work labels Jun 24, 2024
@tclose tclose added the data-ingestion Ingestion of data from edge devices to XNAT label Jun 24, 2024
@tclose
Copy link
Contributor Author

tclose commented Jul 8, 2024 •
edited
Loading

  • Cheapest option is S3 file gateway, no auto-scaling/high availability
  • FSX (Lustre) - good but expensive
  • FS "on-tap" - similar to FSX but cheaper, still more expensive than S3 file-gateway, but might be the best option

@tclose tclose moved this to In Progress in AIS Master Project Jul 8, 2024
@github-actions github-actions bot mentioned this issue Jul 8, 2024
Open
@tclose
Copy link
Contributor Author

tclose commented Aug 5, 2024

Production XNAT is now using S3 File Gateway to store TBP data

@tclose tclose closed this as completed Aug 5, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in AIS Master Project Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fxusyd fxusyd

Labels
data-ingestion Ingestion of data from edge devices to XNAT pipelines story a unit of work
Projects
AIS Master Project
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tclose @fxusyd