Code

feat: new job to backup pins to s3#1844

Code

Open

joshghent wants to merge 69 commits intomainweb3-storage/web3.storage:mainfrom feat/794-copy-pins-to-eipsweb3-storage/web3.storage:feat/794-copy-pins-to-eipsCopy head branch name to clipboard

+7,805-683

Commits

Commits on Oct 12, 2022

• feat: added rough draft of the cron job to copy pins into s3

  

  joshghent

  authored and

  flea89

  committed
  ca658d5

  Copy full SHA for ca658d5

• chore: add aws-sdk

  

  joshghent

  authored and

  flea89

  committed
  cd7a420

  Copy full SHA for cd7a420

• feat: add cron

  

  joshghent

  authored and

  flea89

  committed
  79219bb

  Copy full SHA for 79219bb

• feat: added downloading and uploading of file correctly

  

  joshghent

  authored and

  flea89

  committed
  2cd96c7

  Copy full SHA for 2cd96c7

• chore: added required dependencies

  

  joshghent

  authored and

  flea89

  committed
  48cb64d

  Copy full SHA for 48cb64d

• feat: add new backup_urls column to the psa_pin_requests table

  

  joshghent

  authored and

  flea89

  committed
  55367e7

  Copy full SHA for 55367e7

• feat: finished job to backup files to s3

  

  joshghent

  authored and

  flea89

  committed
  fdb92d3

  Copy full SHA for fdb92d3

• feat: added new test suite for pins backup

  

  joshghent

  authored and

  flea89

  committed
  c910b65

  Copy full SHA for c910b65

• feat: add github action to run cron job

  

  joshghent

  authored and

  flea89

  committed
  bff06de

  Copy full SHA for bff06de

• feat: added cron pins backup script

  

  joshghent

  authored and

  flea89

  committed
  b34dd12

  Copy full SHA for b34dd12

• chore: removed todo

  

  joshghent

  authored and

  flea89

  committed
  14cbd8c

  Copy full SHA for 14cbd8c

• feat: added test mocks

  

  joshghent

  authored and

  flea89

  committed
  ef2600e

  Copy full SHA for ef2600e

• chore: add mocks for export car

  

  joshghent

  authored and

  flea89

  committed
  9e6f035

  Copy full SHA for 9e6f035

• chore: update package-lock, fixed fetch reference

  

  joshghent

  authored and

  flea89

  committed
  ec58dd2

  Copy full SHA for ec58dd2

• feat: swapped to a class tructure

  

  joshghent

  authored and

  flea89

  committed
  0f0cbe5

  Copy full SHA for 0f0cbe5

• chore: update the job to create a new backup class instance

  

  joshghent

  authored and

  flea89

  committed
  d009be1

  Copy full SHA for d009be1

• feat: added more logging, updated query

  

  joshghent

  authored and

  flea89

  committed
  e63a4a1

  Copy full SHA for e63a4a1

• fix: updated test mocks

  

  joshghent

  authored and

  flea89

  committed
  e4e22a0

  Copy full SHA for e4e22a0

• fix: updated to make sql valid

  

  joshghent

  authored and

  flea89

  committed
  381ba65

  Copy full SHA for 381ba65

• Fix issues with updating rows

  flea89

  committed
  9e01bcb

  Copy full SHA for 9e01bcb

• chore: remove redundant argument

  

  joshghent

  authored and

  flea89

  committed
  7271673

  Copy full SHA for 7271673

• chore: update variable names

  

  joshghent

  authored and

  flea89

  committed
  1010da8

  Copy full SHA for 1010da8

• chore: update to use npm for the cron

  

  joshghent

  authored and

  flea89

  committed
  5c0cc8f

  Copy full SHA for 5c0cc8f

• chore: updated type definition to accept a pool not client for pg

  

  joshghent

  authored and

  flea89

  committed
  4b34fdb

  Copy full SHA for 4b34fdb

• chore: remove commented code

  

  joshghent

  authored and

  flea89

  committed
  4b59c7b

  Copy full SHA for 4b59c7b

• feat: added way to not re-upload files we already have

  

  joshghent

  authored and

  flea89

  committed
  1826006

  Copy full SHA for 1826006

• chore: updated package lock

  

  joshghent

  authored and

  flea89

  committed
  e01dda0

  Copy full SHA for e01dda0

• Make linter happier

  flea89

  committed
  3c529e5

  Copy full SHA for 3c529e5

• feat: w3 open <cid> to open cid on w3s.link in browser (#1892)

  Show description for d417922
  

  olizilla

  authored and

  flea89

  committed
  d417922

  Copy full SHA for d417922

• feat: put write to cluster behind a flag (#1785)

  Show description for 7815d72
  

  olizilla

  authored and

  flea89

  committed
  7815d72

  Copy full SHA for 7815d72

• chore: updated package lock

  

  joshghent

  authored and

  flea89

  committed
  396222f

  Copy full SHA for 396222f

• chore: update package lock

  

  joshghent

  authored and

  flea89

  committed
  e3a13da

  Copy full SHA for e3a13da

• feat: pass through the query limit on workflow dispatch

  

  joshghent

  authored and

  flea89

  committed
  6fad50f

  Copy full SHA for 6fad50f

• feat: updated the backup urls column to default to an empty arr

  

  joshghent

  authored and

  flea89

  committed
  5c7bca4

  Copy full SHA for 5c7bca4

• chore: added dagula

  

  joshghent

  authored and

  flea89

  committed
  f07c61b

  Copy full SHA for f07c61b

• feat: added default to empty array

  

  joshghent

  authored and

  flea89

  committed
  36e7e08

  Copy full SHA for 36e7e08

• feat: added default on the init data

  

  joshghent

  authored and

  flea89

  committed
  2d3d56a

  Copy full SHA for 2d3d56a

• feat: updated query to return correct data

  

  joshghent

  authored and

  flea89

  committed
  86b3de6

  Copy full SHA for 86b3de6

• chore: updated return types

  

  joshghent

  authored and

  flea89

  committed
  7ceeaf9

  Copy full SHA for 7ceeaf9

• chore: update package lock

  

  joshghent

  authored and

  flea89

  committed
  7368e13

  Copy full SHA for 7368e13

• chore: remove todo comment

  

  joshghent

  authored and

  flea89

  committed
  e127dc0

  Copy full SHA for e127dc0

• Add fetch to cron pin-backup

  flea89

  committed
  bf8aa7a

  Copy full SHA for bf8aa7a

• Update types

  flea89

  committed
  4abd92f

  Copy full SHA for 4abd92f

• Add Minio to cron

  flea89

  committed
  874830c

  Copy full SHA for 874830c

• Add getS3 to cron utils

  flea89

  committed
  38b96a4

  Copy full SHA for 38b96a4

• update tests to be more e2endy

  flea89

  committed
  6aa93ef

  Copy full SHA for 6aa93ef

• Comment out size limiting

  flea89

  committed
  15fc4fd

  Copy full SHA for 15fc4fd

• more type fixes

  flea89

  committed
  9d2fe99

  Copy full SHA for 9d2fe99

• Use s3 v3 and updats to backup updates

  flea89

  committed
  ae3b044

  Copy full SHA for ae3b044

Commits on Oct 21, 2022

• tidy tests up

  flea89

  committed
  5bc3522

  Copy full SHA for 5bc3522

• Set concurrency to avoid multiple jobs running together

  flea89

  committed
  1777df2

  Copy full SHA for 1777df2

• Use ipfs node id rather than cluster one

  flea89

  committed
  dc431fc

  Copy full SHA for dc431fc

Commits on Oct 24, 2022

• Create a car file for upload and test it holds the expected content

  flea89

  committed
  f621c31

  Copy full SHA for f621c31

• Add abort controller

  flea89

  committed
  a266c32

  Copy full SHA for a266c32

• Merge branch 'main' into feat/794-copy-pins-to-eips

  flea89

  committed
  b7aa9db

  Copy full SHA for b7aa9db

• Align api package with main

  flea89

  committed
  c9e4dbb

  Copy full SHA for c9e4dbb

• Improve logging and more tests

  flea89

  committed
  6e1f121

  Copy full SHA for 6e1f121

• Improve testing and error handling

  flea89

  committed
  d573121

  Copy full SHA for d573121

• Remove only from test

  flea89

  committed
  fd02b6a

  Copy full SHA for fd02b6a

Commits on Oct 25, 2022

• Update logging

  flea89

  committed
  395dff7

  Copy full SHA for 395dff7

• Update node modules

  flea89

  committed
  c58990b

  Copy full SHA for c58990b

• Update setup node action

  flea89

  committed
  52984d6

  Copy full SHA for 52984d6

• Fix dependecies

  flea89

  committed
  694788e

  Copy full SHA for 694788e

• Increase cron timeout

  flea89

  committed
  9dec63f

  Copy full SHA for 9dec63f

Commits on Oct 31, 2022

• Improve logging and remove stale code

  flea89

  committed
  61990b2

  Copy full SHA for 61990b2

• Cache instances of Dagula and use 1 libp2p

  flea89

  committed
  749e20d

  Copy full SHA for 749e20d

• Better instanciation of resources

  flea89

  committed
  be24923

  Copy full SHA for be24923

• Default to quiet logging, but allow for more verbose when running manually

  flea89

  committed
  7a48df2

  Copy full SHA for 7a48df2

• Update cron help text

  flea89

  committed
  8eae8f6

  Copy full SHA for 8eae8f6