Draft: Replace WARC with Obelisk

[fmartingr]

This is a draft to replace WARC with Obelisk in order to know what would be the impact of such a change.

Notes:

• Serving content (not archives) uses the archive in some way. Don't know why, need to investigate
• Old archives would require a migration from the old system (WARC) to the new one (Obelisk) potentially saving up space and being much more confortable to manage since it will create single files.
• Right now storage is file-based, I would move things to database with a cache layer so its easier for us to manage and for users to backup, manage and migrate.

Closes #353

[Katarn]

Right now storage is file-based, I would move things to database with a cache layer so its easier for us to manage and for users to backup, manage and migrate.

I would make this functionality optional. I'm afraid to imagine how large the database dumps will be, they are clearly not intended for such use.

Also it seems that there is not enough setting to indicate in each specific case whether it is necessary to save, for example, JavaScript. The obelisk itself has such an functionality.

[hulb]

Actually I have a need for shiori to save these archive files to object storage like minio :)

[fmartingr]

Actually I have a need for shiori to save these archive files to object storage like minio :)

I believe you already can mount volumes as s3-compatible storage by using docker/k8s?

[Monirzadeh]

Just to not forget maybe possibility of download archive file in api can be helpful for this situation.
DesarrolloAntonio/Shiori-Android-Client#4 (comment)

[stale]

This issue has been automatically marked as stale because it has not had any activity for quite some time.
It will be closed if no further activity occurs.
Thank you for your contributions.

[Monirzadeh]

@fmartingr
why close this? do you send new PR for that?

[fmartingr]

This was very outdated and it has very fee work on it. The PDF archiver refactored most of the logic so I need to start over when that PR is merged.