Skip to content

Commit

Permalink
Add how it works section
Browse files Browse the repository at this point in the history
  • Loading branch information
m-mizutani authored Oct 12, 2024
1 parent 8500def commit ec95c58
Show file tree
Hide file tree
Showing 6 changed files with 379 additions and 6 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
tmp
.vscode

/example/mytest
48 changes: 42 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,56 @@ As a result, security administrators are required to gather logs from multiple s

`hatchery` is a solution designed to address these challenges by collecting data and logs from SaaS services and storing them in object storage. This facilitates log retention and prepares the data for analysis by security administrators.

## How it works

![design overview](./docs/images/design-overview.jpg)

```go
streams := []*hatchery.Stream{
hatchery.NewStream(
// Source: Slack Audit API
slack.New(secret.NewString(os.Getenv("SLACK_TOKEN"))),
// Destination: Google Cloud Storage
gcs.New("mizutani-test"),

// Identifiers
hatchery.WithID("slack-to-gcs"),
hatchery.WithTags("hourly"),
),

hatchery.NewStream(
// Source: 1Password
one_password.New(secret.NewString(os.Getenv("ONE_PASSWORD_TOKEN"))),
// Destination: Amazon S3
s3.New("ap-northeast1", "mizutani-test"),

// Identifiers
hatchery.WithID("1pw-to-s3"),
hatchery.WithTags("daily"),
),
}
```

```go
if err := hatchery.New(streams).CLI(os.Args); err != nil {
panic(err)
}
```

## Documentation

- About Hatchery
- [Overview](docs/README.md)
- [How to Use hatchery](docs/usage.md)
- [How to Develop Hatchery Extension](docs/extension.md)
- Source
- [Slack](source/slack)
- [1Password](source/1password)
- [Falcon Data Replicator](source/falcon_data_replicator)
- [Twilio](source/twilio)
- [Slack](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/source/slack)
- [1Password](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/source/one_password)
- [Falcon Data Replicator](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/source/falcon_data_replicator)
- [Twilio](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/source/twilio)
- Destination
- [Google Cloud Storage](destination/gcs)
- [Amazon S3](destination/s3)
- [Google Cloud Storage](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/destination/gcs)
- [Amazon S3](https://pkg.go.dev/github.com/secmon-as-code/hatchery@main/destination/s3)

## Getting Started

Expand Down
Binary file added docs/images/design-overview.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
68 changes: 68 additions & 0 deletions example/readme/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
module github.com/secmon-as-code/hatchery/example/readme

go 1.23.0

replace github.com/secmon-as-code/hatchery => ../..

require github.com/secmon-as-code/hatchery v0.0.0-20240821222948-f6d88b7f0081

require (
cloud.google.com/go v0.115.1 // indirect
cloud.google.com/go/auth v0.9.4 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
cloud.google.com/go/compute/metadata v0.5.1 // indirect
cloud.google.com/go/iam v1.2.1 // indirect
cloud.google.com/go/storage v1.43.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.30.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.33 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.32 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.13 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.17 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.17 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.61.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.34.9 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.7 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.7 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.30.7 // indirect
github.com/aws/smithy-go v1.20.4 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.13.0 // indirect
github.com/m-mizutani/goerr v0.1.14 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/urfave/cli/v2 v2.27.4 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.55.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.55.0 // indirect
go.opentelemetry.io/otel v1.30.0 // indirect
go.opentelemetry.io/otel/metric v1.30.0 // indirect
go.opentelemetry.io/otel/trace v1.30.0 // indirect
golang.org/x/crypto v0.27.0 // indirect
golang.org/x/net v0.29.0 // indirect
golang.org/x/oauth2 v0.23.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.25.0 // indirect
golang.org/x/text v0.18.0 // indirect
golang.org/x/time v0.6.0 // indirect
google.golang.org/api v0.197.0 // indirect
google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/grpc v1.66.2 // indirect
google.golang.org/protobuf v1.34.2 // indirect
)
Loading

0 comments on commit ec95c58

Please sign in to comment.