Improving Self-Hosting and Removing 3rd Party dependencies.

[Podginator]

The intent of this PR is to improve the Self-Hosting documentation, to provide a working setup to get Omnivore working with Docker and Docker Compose. It intends to, as much as possible, remove third party dependencies and reliance on external infrastructure providers such as GCP.

The aim is to establish feature parity, or near feature parity to the previously hosted service. This includes RSS support, webhook support, email newsletter, and PDF Support.

The list of changes to date is below:

• Create Dockerfile for Queue processing, which is used for parsing articles, alongside asynchronous tasks.

• Update and expose ImageProxy and use the latest version with ARM64 support.

• Create new docker-compose file in self-hosting/docker-compose.

• Provide a minimal .env file to be able to run the service using docker-compose.

• Created a guide for using Cloudflare Tunnels as a way to integrate with a device at your home.

• Create a NGINX configuration for those looking to use NGINX Reverse Proxying for the service.

• Replace use of Google Cloud Storage with Minio an open-source layer compatible with the S3 API that can run on Device.

  • This also allows other services, such as R2 and S3 to be the Storage Provider, if wanted.

• Improvements to content-fetching to minimise instances where articles refused to parse.

  • Also improved to not use puppeteer for some articles, instead relying on raw html.

• Overhaul the way email works, to ensure that there is an open source version. Three options are provided here.

  • Docker Mailserver: A production-ready fullstack but simple containerized mail server. This allows incoming emails to be received, parsed, and then added to Omnivore.
  • Amazon Simple Email Service A service provided by S3 that has a free tier. Allows for receiving of emails to a domain. Guide on how to set up in the Self-hosting readme.
  • Zapier: Used as a way to integrate gmail to hosting. This can be realistically achieved using some of the gmail apis, also.

• Replace pspdfkit - Which required a license and would display the following when using PDFS

  • Have an option for the Native Browser PDF Viewer for PDF Files. This removes the highlight functionality, but is stable.
    
  • Create a new pdf viewer using PDF.js an open source pdf library used as the backing for the PDF viewer in firefox. This option includes near feature parity (highlights, reading progress) with the pspdfkit, but may have some bugs.
    

• Add some additional fixes to parsing articles, such as a Medium Parser, and a Wired parse

• Updated Docker images and software to the latest LTS version of Node (20.12)

To-Do:

• Re-Enable Youtube features - such as extraction of Transcripts.
  • Allow both an AI based feature for this, and a less formatted version.
• Provide a guide on how to get up and running user Kubernetes.
• Provide a guide on how to get up and running with Tailscale.
• Provide a guide on getting email to work with G-Mail without the use of an external server.
• Attempt to provide a lighter-weight queuing system, and removal of Redis/Caching for single-user hosting.

[vercel]

@Podginator is attempting to deploy a commit to the omnivore Team on Vercel.

A member of the Team first needs to authorize it.

[samanthavbarron]

This is awesome, it looks like it's taking shape! I might try it out this weekend.

Are there contributions from the community that you can think of that would be helpful for you?

[lovebes]

Can we use Cloudflare R2 as well for self hosting? What was the decision behind using MinIO?
Asking because R2 is also S3 compatible.

[Podginator]

I'm not actually familiar with R2 - but anything that is S3 Compatible should work. Let me take a look later to see whether or not the Storage Client I built works with it.

Minio was chosen because it can be self-hosted along with the rest of the application. There is a docker image, and it can all run on the same server without relying on anything external.

I'm trying to ensure everything here can be run self-contained without any need for external services.

That said, as with some of the email changes, I am looking into ways to simplify parts of it too, and having some external services is ok with me.

[Mikilio]

To find suitable services, I recommend consulting r/self-hosted.
Love the work so far.

[volker-fr]

S3 is a nice idea, provides various options, including self hosted ones.

How about local storage? This would reduce the required dependencies by one.

[lovebes]

oh wow I didn't know Minio can be self-hosted! That sounds like a good idea.

[Podginator]

S3 is a nice idea, provides various options, including self hosted ones.

How about local storage? This would reduce the required dependencies by one.

The uploads are done via signed-urls, so while local-storage would be feasible it'd require a bit more development work.

[jsifalda]

does it mean i would be able to deploy open-source omnivore to vercel? and be able to use this great app even after their shut-down? 🙏

[RayBB]

If this gets worked out I'll add a template for easy self-hosting with Coolify

[alexkotusenko]

Are you guys planning to add a docker container to self-host Omnivore?

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ Podginator
✅ m1xxos
❌ weeebdev
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[luca-git]

Will this ever happen?

[Podginator]

@luca-git I haven't got the permission to merge, but this branch is a perfectly viable way of getting the application up and running as is. I'll speak with @jacksonh about when the merge should take place.