s3-downloader

Helper utility built in Go to download S3 files without installing the whole AWS CLI. Uses AWS SDK for Go, so supports all its configuration methods (~/.aws/credentials, environment variables, credential servers etc).

Uses Go modules for dependency configuration.

Requirements

• Golang >=1.17.x with Go modules enabled (see development)
• OPTIONAL: make

Installation

Download latest release from GitHub releases

or compile from source

Usage

Usage: ./bin/s3downloader-linux-amd64 <bucket> <prefix> <targetDir>

Logging

This tool logs in a simple JSON format (Voltti's "app-misc" type logs):

{
    "@timestamp": "2020-12-29T12:02:51+02:00",
    "appBuild": "123",
    "appCommit": "22eb5a510ae608615a1487ea2538673c7f84a765",
    "appName": "my-app",
    "env": "dev",
    "exception": "Error", // Optional
    "hostIp": "10.0.0.2",
    "logLevel": "error",
    "message": "Unable to list objects in bucket: my-bucket",
    "stackTrace": "MissingRegion: could not find region configuration", // Optional
    "type": "app-misc",
    "userIdHash": "",
    "version": "1"
}

Configure the output with the following environment variables:

Field	Key	Default
appBuild	APP_BUILD	"local"
appCommit	APP_COMMIT	"HEAD"
appName	APP_NAME	""
env	VOLTTI_ENV or ENV	"local"
hostIp	HOST_IP	""

Development

NOTE: This project uses Go modules, so you should check this repository outside $GOPATH/src to build it.

Install Golang

With homebrew:

sudo brew install go

With apt-get:

sudo apt-get install golang

Or install Golang manually or compile it yourself

Git pre-commit hook

Configure with:

ln -s ../../scripts/pre-commit.sh .git/hooks/pre-commit

Build

make build
# or
make

Releasing

NOTE:: Requires a GitHub Personal access token configured as GITHUB_TOKEN for the CircleCI build (exists for @ci-voltti with scope repo)

Workflow based on: https://circleci.com/blog/publishing-to-github-releases-via-circleci/ and: https://circleci.com/docs/2.0/workflows/#executing-workflows-for-a-git-tag

1. Ensure all changes have been reviewed & merged to master and you have no local changes
2. make release
3. CI creates a GitHub release with artifacts from the tag