Feature: Offloaded Storage/Archive Sync

[tvanroo]

I have a use-case for a sync mode where data is only stored locally temporary (like a cache). Once the data is replicated to StorJ the cache is purged. This makes the mounted StorJ drive a great place for archiving large amounts of data that are no longer needed on a day-to-day basis and allowing the local storage to be freed up.

[MeijeSibbel]

Maybe we can add this as a option as dropbox does for example, so you can select a drive (or a bunch of files), right click and --> import to goobox.

[kaloyan-raev]

This is an interesting use case. Technically, this kind of sync logic is much easier to implement than the two-way sync I am currently working on.

Perhaps, we can provide users to choose from a set of sync modes:

• Box mode - the current two-way sync mode. You put a file in the box and you expect it will be still there next time you check.
• Sink mode - the proposed one here. You put a file in the sink and you expect it is gone once it sinks into the Storj bucket.

@tvanroo I have a few questions to understand better the use case:

• How do you plan to access the files after they are transferred to Storj and respectively deleted from the local storage? Via another app (like Filezilla)?
• Could you give some real life examples of data that make sense to transfer to Storj this way?
• What should happen if a file is copied to the local Goobox folder, but a file with the same name already exists in the Storj bucket?
  -- Overwrite the file in the Storj bucket with the new content
  -- Ignore the new local file (even deleted it) without transferring it to Storj
  -- Prompt a dialog to the user to choose?

[tvanroo]

@MeijeSibbel It would be cool to be able to right click on a folder and select "offload this folder." I think people can relate to freeing up space but not wanting to lose data. It would be very cool if, once offloaded, a shortcut to the new location of the data was created. (That shortcut redirecting to the mounted drive/path I describe below).

@kaloyan-raev I was thinking the files would still present in the Windows Explorer Window, likely as a mounted drive, likely with some sort of configurable cache behind the scene). Configuring the cache path and size are important for my Terminal Server use case) I've seen this done by Mountain Duck (with FTP Storage fro example).

For my work we host desktop environments (Terminal Servers) for tens of thousands of users, the local drives are fast, redundant, backed up and expensive. Many clients have to keep old records for compliance and legal reasons but don't actually need to access them unless a lawsuit/audit requires it. In this situation the ability to use StorJ as an Archive destination (without losing viability and without adding complexity for the user) is huge. They can achieve a reduction in storage cost of over 20x.

Another use case... Build a decentralized alternative to Google Drive. I can run a laptop with a small 128GB drive but still have cloud access (as a mounted folder) to all my stuff that won't fit on the local HD.

I think you default to prompt the user but allow for that setting to be changed.

[kaloyan-raev]

@tvanroo

I was thinking the files would still present in the Windows Explorer Window, likely as a mounted drive, likely with some sort of configurable cache behind the scene). Configuring the cache path and size are important for my Terminal Server use case) I've seen this done by Mountain Duck (with FTP Storage fro example).

This would require implementing a file system driver for Windows. This can be done in C only and using Windows-specific API, i.e. the work cannot be reused for Linux and macOS. Technically, it is a completely different project that the current sync app. If done properly, it can be a really great piece of work that can cover all the scenarios you've described.

But... without exaggerating, implementing such file system driver is 100x more expensive (in terms of development time and efforts) compared to the "sink folder" I suggested above. The sink folder can be easily implemented in Java reusing the current sync app, and why not even integrating with it. It can work on all major OS types: Windows, Linux and macOS.

Of course, the sink folder would not cover all the described scenarios above, but it will cover the business scenario:

For my work we host desktop environments (Terminal Servers) for tens of thousands of users, the local drives are fast, redundant, backed up and expensive. Many clients have to keep old records for compliance and legal reasons but don't actually need to access them unless a lawsuit/audit requires it. In this situation the ability to use StorJ as an Archive destination (without losing viability and without adding complexity for the user) is huge. They can achieve a reduction in storage cost of over 20x.

So, in the end, it is a matter of priorities and budget :)

[MeijeSibbel]

Alternatively we can still use a sync folder but also implement the offloading feature, maybe we can create some way to list these offloaded files in the sync folder (maybe a specific directory) but only allow users to download them (as they are not stored locally). I agree with Kaloyan 100%, a mounted drive is allot more work and outside the scope right now, first we have to get a working product asap, all other features can come later.

[kaloyan-raev]

For reference, Dropbox offers a paid Smart Sync feature to avoid aggressively sync every file from the cloud to the local drive: https://www.dropbox.com/smartsync

[MeijeSibbel]

Good catch

With Smart Sync, you can make files and folders online-only. Online-only files use just a small amount of hard drive space to store information like the file name, size, and thumbnail, but the full contents of the files are stored securely in the Dropbox cloud. You’ll always see your online-only files on your computer, so when you need them, you can open them with just a click.

[kaloyan-raev]

In order to implement it in Goobox, we need to research how to add virtual files and folders in the Goobox sync folder.