Skip to content
This repository has been archived by the owner on Oct 29, 2024. It is now read-only.

Commit

Permalink
Better readme, changing package name
Browse files Browse the repository at this point in the history
from .backups to .cgov-util
  • Loading branch information
jadudm committed Mar 2, 2024
1 parent 4c82f8d commit 816e20c
Show file tree
Hide file tree
Showing 14 changed files with 50 additions and 37 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build_go.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go

name: Build gov.gsa.fac.backups
name: Build gov.gsa.fac.cgov-util

env:
GO_VERSION: '1.22'
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# .github/workflows/release.yaml

name: Release gov.gsa.fac.backups
name: Release gov.gsa.fac.cgov-util

on:
release:
Expand All @@ -22,5 +22,5 @@ jobs:
goos: linux
goarch: amd64
goversion: "https://go.dev/dl/go1.22.0.linux-amd64.tar.gz"
binary_name: "gov.gsa.fac.backups"
binary_name: "gov.gsa.fac.cgov-util"
extra_files: LICENSE README.md
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1 @@
gov.gsa.fac.backups
gov.gsa.fac.cgov-util
33 changes: 26 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,47 @@
## while developing/testing

```
go run main.go clone-db-to-db
go run main.go
```

## to build

```
go build
./build.sh
```

## to run
## Usage: clone

```
./gov.gsa.fac.backups
gov.gsa.fac.cgov-util clone --source-db <name> --destination-db <name>
```

to see the options, and
This command clones one DB to another by piping STDOUT from `pg_dump` into the STDIN of `psql`, with the correct connection/credential parameters for each command.

When run localling (assuming `ENV` is set to `LOCAL`) it will read a `config.json` from the directory `$HOME/.fac/config.json` (or, from `config.json` in the same folder as the application). This file should look like a `VCAP_SERVICES` variable that would be encountered in the Cloud Foundry/cloud.gov environment.

When run in the cloud.gov environment (where `ENV` is anything other than `LOCAL` or `TESTING`), it will look at `$VCAP_SERVICES`, look in the `aws-rds` key, and look up the DB credentials by the friendly name provided on the command line. By this, if your brokered DB is called `fac-db`, this will then populate the credentials (internally) with the brokered DB name, password, URI, etc. in order to correctly `pg_dump` from one and, using another set of credentials, stream the data into another.

This does *not* guarantee a perfect backup. It *does* do a rapid snapshot at a moment in time, without requiring the application to write any files to the local filesystem within a container. (On cloud.gov, this is limited to ~6GB, which makes dumping and loading DBs difficult.)

## Usage: bucket

```
./gov.gsa.fac.backups clone-db-to-db
gsa.gov.fac.cgov-util bucket --source-db <name> --destination-bucket <name>
```

to backup a source DB to a destination, based on `config.yaml` settings.
Similar to above, but this pipes a `pg_dump` to `s3 copy`.

For now, this writes to the key `s3://<bucket>/backups/<name>-<db-name>.dump`

This wants to be improved.

The purpose here is to (again) dump a database to a storage location without touching the local (containerized) filesystem. It uses friendly names, again, to look up the credentials for both the RDS database and brokered S3 in order to stream a DB dump to S3. (In theory, S3 does multipart uploads, so you should end up with a single file, up to 5TB in size, for your dump.)

When running locally, this assumes `minio` is running as a stand-in for S3, and is specified as a `user-specified` service in the (local, bogus) VCAP_SERVICES config.

(An example `config.json` is in this repository, and a more complete file in `internal/vcap/vcap_test.go`).


## assumptions

Expand Down
6 changes: 3 additions & 3 deletions cmd/bucket.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import (
"golang.org/x/exp/slices"

"github.com/spf13/cobra"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/pipes"
vcap "gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/pipes"
vcap "gov.gsa.fac.cgov-util/internal/vcap"
)

func bucket_local(source_creds *vcap.CredentialsRDS, up vcap.UserProvidedCredentials) {
Expand Down
4 changes: 2 additions & 2 deletions cmd/check.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ import (
"os"

"github.com/spf13/cobra"
"gov.gsa.fac.backups/internal/logging"
vcap "gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
vcap "gov.gsa.fac.cgov-util/internal/vcap"
)

func get_row_count(creds *vcap.CredentialsRDS, table string) int {
Expand Down
6 changes: 3 additions & 3 deletions cmd/clone.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ import (
"os"

"github.com/spf13/cobra"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/pipes"
vcap "gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/pipes"
vcap "gov.gsa.fac.cgov-util/internal/vcap"

_ "github.com/lib/pq"
)
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module gov.gsa.fac.backups
module gov.gsa.fac.cgov-util

go 1.20

Expand Down
6 changes: 2 additions & 4 deletions internal/pipes/mc.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ import (

"github.com/bitfield/script"
"github.com/google/uuid"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/vcap"
)

// https://bitfieldconsulting.com/golang/scripting
Expand Down Expand Up @@ -41,8 +41,6 @@ func Mc(in_pipe *script.Pipe, upc vcap.UserProvidedCredentials, prefix string, s
}
// Combine the slice for printing and execution.
combined := strings.Join(cmd[:], " ")
// This will log the password...
logging.Logger.Printf("BACKUPS Running `%s`\n", combined)
logging.Logger.Printf("BACKUPS mc targeting %s", prefix)
return in_pipe.Exec(combined)
}
6 changes: 2 additions & 4 deletions internal/pipes/pg_dump.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"strings"

"github.com/bitfield/script"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/vcap"
)

// https://bitfieldconsulting.com/golang/scripting
Expand All @@ -31,8 +31,6 @@ func PG_Dump(creds *vcap.CredentialsRDS) *script.Pipe {
}
// Combine the slice for printing and execution.
combined := strings.Join(cmd[:], " ")
// This will log the password...
logging.Logger.Printf("BACKUPS Running `%s`\n", combined)
logging.Logger.Printf("BACKUPS pg_dump targeting %s", creds.DB_Name)
return script.Exec(combined)
}
4 changes: 2 additions & 2 deletions internal/pipes/psql.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"strings"

"github.com/bitfield/script"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/vcap"
)

func Psql(in_pipe *script.Pipe, creds *vcap.CredentialsRDS) *script.Pipe {
Expand Down
6 changes: 2 additions & 4 deletions internal/pipes/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"strings"

"github.com/bitfield/script"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/internal/logging"
"gov.gsa.fac.cgov-util/internal/vcap"
)

// https://bitfieldconsulting.com/golang/scripting
Expand All @@ -26,8 +26,6 @@ func S3(in_pipe *script.Pipe, up *vcap.CredentialsS3, prefix string, source_db s

// Combine the slice for printing and execution.
combined := strings.Join(cmd[:], " ")
// This will log the password...
logging.Logger.Printf("BACKUPS Running `%s`\n", combined)
logging.Logger.Printf("BACKUPS s3 targeting %s", prefix)
return in_pipe.Exec(combined)
}
2 changes: 1 addition & 1 deletion internal/vcap/vcap.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (

"github.com/pkg/errors"
"github.com/spf13/viper"
"gov.gsa.fac.backups/internal/logging"
"gov.gsa.fac.cgov-util/internal/logging"
)

type CredentialsRDS struct {
Expand Down
4 changes: 2 additions & 2 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
"golang.org/x/exp/slices"

"github.com/spf13/viper"
"gov.gsa.fac.backups/cmd"
"gov.gsa.fac.backups/internal/vcap"
"gov.gsa.fac.cgov-util/cmd"
"gov.gsa.fac.cgov-util/internal/vcap"
)

var SHA1 string
Expand Down

0 comments on commit 816e20c

Please sign in to comment.