Web-file-cache

Web file cache service based on GitHub actions

Technologies

• Crawler: web-fetch (Puppeteer based)
• OSS service: Cloudflare R2

Usage

Setup repository

1. Install GitHub apps in your organization or account:

   1. Probot settings: set up Issue labels & Pull Request rules

2. Click the Use this template button on the top of this GitHub repository's home page, then create your own repository in the app-installed namespace above

3. Set Cloudflare variables is .github/workflows/crawler.yml as Repository secrets

Manual cache

https://github.com/idea2app/Web-file-cache/issues/new?template=crawler.yml

Automatic cache

$URL = "https://example.com/test.html"

curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -d '{"title":"File title","body":"### URL\n\n$URL","labels":["crawler"]}' \
  https://api.github.com/repos/idea2app/Web-file-cache/issues