Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stable NR Reporting #38

Closed
wants to merge 9 commits into from
Closed

Stable NR Reporting #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
be3c164
fix: jq needs to be installed in the container, not the host
nickumia-reisys Apr 30, 2022
77d0a8b
fix: try without sudo
nickumia-reisys Apr 30, 2022
1e7827c
fix: open /root/.cf/config.json: no such file or directory
nickumia-reisys Apr 30, 2022
5f314a9
test: multiline one line command
nickumia-reisys Apr 30, 2022
212036b
test: multiline command is not possible
nickumia-reisys Apr 30, 2022
9431801
cleanup: add back dependencies
nickumia-reisys Apr 30, 2022
560f85a
new: update logstash from 7.16.3 to 7.17.3
nickumia-reisys May 1, 2022
e123dae
new: configure the correct NR uri/key pair
nickumia-reisys May 1, 2022
8cc0d85
fix: logstash is different of logstash-oss :/
nickumia-reisys May 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,10 @@ jobs:
steps:
- name: checkout
uses: actions/checkout@v2
- name: Install Dependencies
run: sudo apt-get install -y jq
- name: drain-staging-space
uses: cloud-gov/cg-cli-tools@main
with:
command: cf install-plugin -f -r CF-Community "drains" && ./create-space-drain.sh
command: apt-get install -y jq && cf install-plugin -f -r CF-Community "drains" && mkdir -p /root/.cf/ && touch /root/.cf/config.json && ./create-space-drain.sh
cf_org: gsa-datagov
cf_space: staging
cf_username: ${{secrets.CF_SERVICE_USER}}
Expand All @@ -122,12 +120,10 @@ jobs:
steps:
- name: checkout
uses: actions/checkout@v2
- name: Install Dependencies
run: sudo apt-get install -y jq
- name: drain-prod-space
uses: cloud-gov/cg-cli-tools@main
with:
command: cf install-plugin -f -r CF-Community "drains" && ./create-space-drain.sh
command: apt-get install -y jq && cf install-plugin -f -r CF-Community "drains" && mkdir -p /root/.cf/ && touch /root/.cf/config.json && ./create-space-drain.sh
cf_org: gsa-datagov
cf_space: prod
cf_username: ${{secrets.CF_SERVICE_USER}}
Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
logstash-installation:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o logstash/awscliv2.zip
curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.3-linux-x86_64.tar.gz -o logstash/logstash-oss-7.16.3-linux-x86_64.tar.gz
docker run -v $${PWD}:$${PWD} -w $${PWD} --rm docker.elastic.co/logstash/logstash-oss:7.16.3 bin/prepare-offline-logstash
curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.17.3-linux-x86_64.tar.gz -o logstash/logstash-oss-7.17.3-linux-x86_64.tar.gz
docker run -v $${PWD}:$${PWD} -w $${PWD} --rm docker.elastic.co/logstash/logstash-oss:7.17.3 bin/prepare-offline-logstash
6 changes: 3 additions & 3 deletions logstash/.profile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ function parse_vcap_services () {
parse_vcap_services

echo "Unpacking logstash..."
tar xzvf logstash-oss-7.16.3-linux-x86_64.tar.gz > /dev/null 2>&1 && \
rm logstash-oss-7.16.3-linux-x86_64.tar.gz
export LS_HOME="$PWD/logstash-7.16.3"
tar xzvf logstash-oss-7.17.3-linux-x86_64.tar.gz > /dev/null 2>&1 && \
rm logstash-oss-7.17.3-linux-x86_64.tar.gz
export LS_HOME="$PWD/logstash-7.17.3"

echo "Installing logstash plugins..."
"$LS_HOME"/bin/logstash-plugin install file://"$PWD"/plugins.zip
Expand Down
61 changes: 30 additions & 31 deletions logstash/logstash.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,50 +63,49 @@ filter{
# }
# mutate { add_field => { "[@metadata][target_index]" => "index-%{+YYYY.MM}" } }

mutate {
add_field => { "[@metadata][NEWRELIC_KEY]" => "${NEWRELIC_LICENSE_KEY:notpresent}" }
add_field => { "[@metadata][AWS_S3_PROXY]" => "${AWS_S3_PROXY:notpresent}" }
}
# mutate {
# add_field => { "[@metadata][AWS_S3_PROXY]" => "${AWS_S3_PROXY:notpresent}" }
# }

}

output {
# Explicit support for environment variable expansion in conditionals is not yet supported, though it's very close:

# Explicit support for environment variable expansion in conditionals is not yet supported, though it's very close:
# https://github.com/elastic/logstash/pull/13370
#
# There's a clever workaround described here...
# https://github.com/elastic/logstash/issues/5115#issuecomment-214283550
# https://github.com/elastic/logstash/issues/5115#issuecomment-279552849
#
# See the mutate filter above that sets this value
if [@metadata][NEWRELIC_KEY] != "notpresent" {
if "${NEWRELIC_LICENSE_KEY}" != "" {
newrelic {
base_uri => "https://log-api.newrelic.com/log/v1"
license_key => "[@metadata][NEWRELIC_KEY]"
base_uri => "${NEWRELIC_LOG_URI}"
license_key => "${NEWRELIC_LICENSE_KEY}"
}
}

if [@metadata][AWS_S3_PROXY] != "notpresent" {
s3 {
access_key_id => "${AWS_ACCESS_KEY_ID:yourawsaccesskeyid}"
secret_access_key => "${AWS_SECRET_ACCESS_KEY:yourawsaccesskeysecret}"
region => "${AWS_REGION:yourawsregion}"
bucket => "${AWS_BUCKET:yourawsbucket}"
endpoint => "${AWS_ENDPOINT:https://yourawsendpoint}"
proxy_uri => "[@metadata][AWS_S3_PROXY]"
validate_credentials_on_root_bucket => "false"
additional_settings => { "force_path_style" => true }
}
} else {
s3 {
access_key_id => "${AWS_ACCESS_KEY_ID:yourawsaccesskeyid}"
secret_access_key => "${AWS_SECRET_ACCESS_KEY:yourawsaccesskeysecret}"
region => "${AWS_REGION:yourawsregion}"
bucket => "${AWS_BUCKET:yourawsbucket}"
endpoint => "${AWS_ENDPOINT:https://yourawsendpoint}"
validate_credentials_on_root_bucket => "false"
additional_settings => { "force_path_style" => true }
}
}
# if [@metadata][AWS_S3_PROXY] != "notpresent" {
# s3 {
# access_key_id => "${AWS_ACCESS_KEY_ID:yourawsaccesskeyid}"
# secret_access_key => "${AWS_SECRET_ACCESS_KEY:yourawsaccesskeysecret}"
# region => "${AWS_REGION:yourawsregion}"
# bucket => "${AWS_BUCKET:yourawsbucket}"
# endpoint => "${AWS_ENDPOINT:https://yourawsendpoint}"
# proxy_uri => "[@metadata][AWS_S3_PROXY]"
# validate_credentials_on_root_bucket => "false"
# additional_settings => { "force_path_style" => true }
# }
# } else {
# s3 {
# access_key_id => "${AWS_ACCESS_KEY_ID:yourawsaccesskeyid}"
# secret_access_key => "${AWS_SECRET_ACCESS_KEY:yourawsaccesskeysecret}"
# region => "${AWS_REGION:yourawsregion}"
# bucket => "${AWS_BUCKET:yourawsbucket}"
# endpoint => "${AWS_ENDPOINT:https://yourawsendpoint}"
# validate_credentials_on_root_bucket => "false"
# additional_settings => { "force_path_style" => true }
# }
# }
}
3 changes: 2 additions & 1 deletion manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ applications:
- ((app_name))-s3
- ((app_name))-secrets
env:
NEWRELIC_LICENSE_KEY: ((logstash_newrelic_license_key))
NEWRELIC_LICENSE_KEY: ((logstash_newrelic_license_key))
NEWRELIC_LOG_URI: ((logstash_newrelic_log_uri))
1 change: 1 addition & 0 deletions vars.management-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ https_proxy:
# Optional license key for delivering logs to New Relic
# See https://github.com/newrelic/logstash-output-plugin
logstash_newrelic_license_key:
logstash_newrelic_log_uri: https://gov-log-api.newrelic.com/log/v1
1 change: 1 addition & 0 deletions vars.management.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ https_proxy:
# Optional license key for delivering logs to New Relic
# See https://github.com/newrelic/logstash-output-plugin
logstash_newrelic_license_key:
logstash_newrelic_log_uri: https://gov-log-api.newrelic.com/log/v1