feat: add tracing to collection API #236

jpetto · 2024-11-08T20:59:23Z

Goal

add tracing to collection API (take 2)

uses pocket's OTLP collector endpoint
sends traces AND logs to GCP for easy viewing
updates sentry on all ECS services to version 8
- version 8 has most of our integrations built in - yay!
removed some legacy infra code

This has been deployed to dev - you can see traces in GCP here.

Implementation Decisions

instead of sending to AWS X-Ray (which was buggy and memory intensive), sends traces and logs to a standalone ECS service, which in turn sends traces and logs on to GCP for better monitoring.
integrating feature flags/unleash so we can turn tracing on or off, and set the sample rate for traces. also gives us the ability to use feature flags for other things!
greatly reduced logging levels application wide. we don't need to log debug level events, even in dev. (this can be re-enabled in dev easily if needs be.)

Deployment steps

Deployed to dev?
Secrets - created!

References

JIRA ticket:

https://mozilla-hub.atlassian.net/browse/MC-1588

github-actions · 2024-11-08T21:00:39Z

Plan Result (prospect-translation-lambda-cdk-production)

CI link

Plan: 0 to add, 1 to change, 0 to destroy.

Update
- aws_lambda_function.translation-lambda_translation-sqs-lambda_B9BDF6BA

Change Result (Click me)

  # aws_lambda_function.translation-lambda_translation-sqs-lambda_B9BDF6BA will be updated in-place
  ~ resource "aws_lambda_function" "translation-lambda_translation-sqs-lambda_B9BDF6BA" {
        id                             = "ProspectAPI-Prod-Sqs-Translation-Function"
        tags                           = {
            "app_code"       = "content"
            "component_code" = "content-prospectapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "ProspectAPI-Sqs-Translation"
        }
        # (22 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"                      = (sensitive value)
                # (4 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

⚠️ Errors

add a label prospect-translation-lambda-cdk-production/add-or-update: POST https://api.github.com/repos/Pocket/content-monorepo/issues/236/labels: 422 Validation Failed [{Resource:Label Field:name Code:invalid Message:}]

github-actions · 2024-11-08T21:00:46Z

Plan Result (corpus-scheduler-lambda-cdk-production)

CI link

Plan: 0 to add, 1 to change, 0 to destroy.

Update
- aws_lambda_function.corpus-scheduler-sqs-lambda_F2ECDF9F

Change Result (Click me)

  # aws_lambda_function.corpus-scheduler-sqs-lambda_F2ECDF9F will be updated in-place
  ~ resource "aws_lambda_function" "corpus-scheduler-sqs-lambda_F2ECDF9F" {
        id                             = "CorpusSchedulerLambda-Prod-SQS-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:CorpusSchedulerLambda-Prod-SQS-Function:169" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:CorpusSchedulerLambda-Prod-SQS-Function:169/invocations" -> (known after apply)
        tags                           = {
            "app_code"       = "content"
            "component_code" = "content-corpusschedulerlambda"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "CorpusSchedulerLambda"
        }
      ~ version                        = "169" -> (known after apply)
        # (20 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"                          = (sensitive value)
                # (7 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

⚠️ Errors

add a label corpus-scheduler-lambda-cdk-production/add-or-update: POST https://api.github.com/repos/Pocket/content-monorepo/issues/236/labels: 422 Validation Failed [{Resource:Label Field:name Code:invalid Message:}]

github-actions · 2024-11-08T21:00:56Z

Plan Result (prospect-api-cdk-production)

CI link

Plan: 0 to add, 3 to change, 0 to destroy.

Update
- aws_dynamodb_table.dynamodb_prospects_dynamodb_table_9854E41E
- aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6
- aws_lambda_function.bridge-lambda_bridge-sqs-lambda_343B543A

Change Result (Click me)

  # data.aws_iam_policy_document.application_ecs_service_ecs-iam_data-ecs-task-role-policy_090CC3AD will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "application_ecs_service_ecs-iam_data-ecs-task-role-policy_090CC3AD" {
      + id            = (known after apply)
      + json          = (known after apply)
      + minified_json = (known after apply)
      + version       = "2012-10-17"

      + statement {
          + actions   = [
              + "dynamodb:BatchGet*",
              + "dynamodb:DescribeTable",
              + "dynamodb:Get*",
              + "dynamodb:Query",
              + "dynamodb:Scan",
              + "dynamodb:UpdateItem",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
            ]
        }
      + statement {
          + actions   = [
              + "s3:*",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-prospectapi-prod-images",
              + "arn:aws:s3:::pocket-prospectapi-prod-images/*",
            ]
        }
      + statement {
          + actions   = [
              + "events:PutEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:events:us-east-1:996905175585:event-bus/PocketEventBridge-Prod-Shared-Event-Bus",
            ]
        }
      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
              + "logs:CreateLogStream",
              + "logs:DescribeLogGroups",
              + "logs:DescribeLogStreams",
              + "logs:PutLogEvents",
              + "xray:GetSamplingRules",
              + "xray:GetSamplingStatisticSummaries",
              + "xray:GetSamplingTargets",
              + "xray:PutTelemetryRecords",
              + "xray:PutTraceSegments",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
    }

  # aws_dynamodb_table.dynamodb_prospects_dynamodb_table_9854E41E will be updated in-place
  ~ resource "aws_dynamodb_table" "dynamodb_prospects_dynamodb_table_9854E41E" {
        id                          = "PROAPI-Prod-Prospects"
        name                        = "PROAPI-Prod-Prospects"
        tags                        = {
            "app_code"       = "content"
            "component_code" = "content-prospectapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "ProspectAPI"
        }
        # (9 unchanged attributes hidden)

      - global_secondary_index {
          - hash_key           = "scheduledSurfaceGuid" -> null
          - name               = "scheduledSurfaceGuid-prospectType" -> null
          - non_key_attributes = [] -> null
          - projection_type    = "ALL" -> null
          - range_key          = "prospectType" -> null
          - read_capacity      = 0 -> null
          - write_capacity     = 0 -> null
        }
      + global_secondary_index {
          + hash_key           = "scheduledSurfaceGuid"
          + name               = "scheduledSurfaceGuid-prospectType"
          + non_key_attributes = []
          + projection_type    = "ALL"
          + range_key          = "prospectType"
          + read_capacity      = 5
          + write_capacity     = 5
        }

        # (5 unchanged blocks hidden)
    }

  # aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6 will be updated in-place
  ~ resource "aws_iam_policy" "application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6" {
        id               = "arn:aws:iam::996905175585:policy/ProspectAPI-Prod-TaskRolePolicy"
        name             = "ProspectAPI-Prod-TaskRolePolicy"
      ~ policy           = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "dynamodb:UpdateItem",
                          - "dynamodb:Scan",
                          - "dynamodb:Query",
                          - "dynamodb:Get*",
                          - "dynamodb:DescribeTable",
                          - "dynamodb:BatchGet*",
                        ]
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
                        ]
                    },
                  - {
                      - Action   = "s3:*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:s3:::pocket-prospectapi-prod-images/*",
                          - "arn:aws:s3:::pocket-prospectapi-prod-images",
                        ]
                    },
                  - {
                      - Action   = "events:PutEvents"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:events:us-east-1:996905175585:event-bus/PocketEventBridge-Prod-Shared-Event-Bus"
                    },
                  - {
                      - Action   = [
                          - "xray:PutTraceSegments",
                          - "xray:PutTelemetryRecords",
                          - "xray:GetSamplingTargets",
                          - "xray:GetSamplingStatisticSummaries",
                          - "xray:GetSamplingRules",
                          - "logs:PutLogEvents",
                          - "logs:DescribeLogStreams",
                          - "logs:DescribeLogGroups",
                          - "logs:CreateLogStream",
                          - "logs:CreateLogGroup",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags             = {
            "app_code"       = "content"
            "component_code" = "content-prospectapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "ProspectAPI"
        }
        # (5 unchanged attributes hidden)
    }

  # aws_lambda_function.bridge-lambda_bridge-sqs-lambda_343B543A will be updated in-place
  ~ resource "aws_lambda_function" "bridge-lambda_bridge-sqs-lambda_343B543A" {
        id                             = "ProspectAPI-Prod-Sqs-Bridge-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Bridge-Function:213" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Bridge-Function:213/invocations" -> (known after apply)
        tags                           = {
            "app_code"       = "content"
            "component_code" = "content-prospectapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "ProspectAPI"
        }
      ~ version                        = "213" -> (known after apply)
        # (20 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"                  = (sensitive value)
                # (4 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

github-actions · 2024-11-08T21:01:13Z

Plan Result (collection-api-cdk-production)

CI link

⚠️ Resource Deletion will happen

This plan contains resource delete operation. Please check the plan result very carefully!

Plan: 1 to add, 2 to change, 1 to destroy.

Update
- aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-execution-role-policy_2D469A77
- aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6
Replace
- aws_ecs_task_definition.application_ecs_service_ecs-task_461CC9D4

Change Result (Click me)

  # aws_ecs_task_definition.application_ecs_service_ecs-task_461CC9D4 must be replaced
-/+ resource "aws_ecs_task_definition" "application_ecs_service_ecs-task_461CC9D4" {
      ~ arn                      = "arn:aws:ecs:us-east-1:996905175585:task-definition/CollectionAPI-Prod:938" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:996905175585:task-definition/CollectionAPI-Prod" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [
              ~ {
                  ~ environment            = [
                        # (1 unchanged element hidden)
                        {
                            name  = "EVENT_BUS_NAME"
                            value = "PocketEventBridge-Prod-Shared-Event-Bus"
                        },
                      + {
                          + name  = "LOG_LEVEL"
                          + value = "info"
                        },
                        {
                            name  = "NODE_ENV"
                            value = "production"
                        },
                      + {
                          + name  = "OTLP_COLLECTOR_URL"
                          + value = "https://otel-collector.readitlater.com:443"
                        },
                    ]
                    name                   = "app"
                  ~ secrets                = [
                        # (1 unchanged element hidden)
                        {
                            name      = "SENTRY_DSN"
                            valueFrom = "arn:aws:ssm:us-east-1:996905175585:parameter/CollectionAPI/Prod/SENTRY_DSN"
                        },
                      + {
                          + name      = "UNLEASH_ENDPOINT"
                          + valueFrom = "arn:aws:ssm:us-east-1:996905175585:parameter/Shared/Prod/UNLEASH_ENDPOINT"
                        },
                      + {
                          + name      = "UNLEASH_KEY"
                          + valueFrom = "arn:aws:secretsmanager:us-east-1:996905175585:secret:CollectionAPI/Prod/UNLEASH_KEY"
                        },
                    ]
                    # (9 unchanged attributes hidden)
                },
            ] # forces replacement
        )
      ~ id                       = "CollectionAPI-Prod" -> (known after apply)
      ~ revision                 = 938 -> (known after apply)
        tags                     = {
            "app_code"       = "pocket"
            "component_code" = "pocket-collectionapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "CollectionAPI"
        }
        # (10 unchanged attributes hidden)
    }

  # aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-execution-role-policy_2D469A77 will be updated in-place
  ~ resource "aws_iam_policy" "application_ecs_service_ecs-iam_ecs-task-execution-role-policy_2D469A77" {
        id               = "arn:aws:iam::996905175585:policy/CollectionAPI-Prod-TaskExecutionRolePolicy"
        name             = "CollectionAPI-Prod-TaskExecutionRolePolicy"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      - Sid      = ""
                        # (3 unchanged attributes hidden)
                    },
                  ~ {
                      ~ Resource = [
                          + "arn:aws:ssm:us-east-1:996905175585:parameter/Shared/Prod/*",
                          + "arn:aws:ssm:us-east-1:996905175585:parameter/Shared/Prod",
                            "arn:aws:ssm:us-east-1:996905175585:parameter/CollectionAPI/Prod/*",
                            # (1 unchanged element hidden)
                        ]
                      - Sid      = ""
                        # (2 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {
            "app_code"       = "pocket"
            "component_code" = "pocket-collectionapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "CollectionAPI"
        }
        # (5 unchanged attributes hidden)
    }

  # aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6 will be updated in-place
  ~ resource "aws_iam_policy" "application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6" {
        id               = "arn:aws:iam::996905175585:policy/CollectionAPI-Prod-TaskRolePolicy"
        name             = "CollectionAPI-Prod-TaskRolePolicy"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                    # (1 unchanged element hidden)
                    {
                        Action   = "events:PutEvents"
                        Effect   = "Allow"
                        Resource = "arn:aws:events:us-east-1:996905175585:event-bus/PocketEventBridge-Prod-Shared-Event-Bus"
                    },
                  ~ {
                      ~ Action   = [
                          - "xray:PutTraceSegments",
                          - "xray:PutTelemetryRecords",
                          - "xray:GetSamplingTargets",
                          - "xray:GetSamplingStatisticSummaries",
                          - "xray:GetSamplingRules",
                            "logs:PutLogEvents",
                            # (4 unchanged elements hidden)
                        ]
                        # (2 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {
            "app_code"       = "pocket"
            "component_code" = "pocket-collectionapi"
            "env_code"       = "prod"
            "environment"    = "Prod"
            "service"        = "CollectionAPI"
        }
        # (5 unchanged attributes hidden)
    }

Plan: 1 to add, 2 to change, 1 to destroy.

Changes to Outputs:
  ~ ecs-task-arn           = "arn:aws:ecs:us-east-1:996905175585:task-definition/CollectionAPI-Prod:938" -> (known after apply)

bassrock · 2024-11-12T17:16:44Z

README.md

+2. Toggle the `default` environment to "On" in the left hand "Enabled in environments" box
+3. Expand the `default` environment in the main, right-hand panel
+4. Click the ✎ pencil icon to edit the "Gradual rollout" strategy
+5. Move the "Rollout" slider to the percentage of traces you want to collect


Rollout should always be 100%, and you want to edit the variant within the rollouts number value to a value between 0-1 to represent the sampling percent.

katerinachinnappan · 2024-11-13T01:17:44Z

README.md


- When adding a new aws-sdk, pin it to the version used throughout the monorepo.
- When upgrading aws-sdk, upgrade it consistently throughout the monorepo.
+The syncpack config can be found in the `./syncpackrc` file.


thanks for the extensive documentation 🙏

katerinachinnappan · 2024-11-13T01:18:21Z

README.md

+
+To disable tracing on a service, simply toggle the `default` environment to "Off".
+
+### Local Tracing


katerinachinnappan

🚀 🔥

- uses pocket's OTLP collector endpoint - sends traces AND logs to GCP for easy viewing

github-actions bot added the prospect-api-cdk-production/add-or-update label Nov 8, 2024

github-actions bot added curated-corpus-api-cdk-production/no-changes collection-api-cdk-production/destroy labels Nov 8, 2024

jpetto force-pushed the MC-1588-add-otlp-to-collection-api branch 3 times, most recently from 70e9657 to 753cd21 Compare November 8, 2024 21:35

github-actions bot deployed to curated-corpus-api-dev November 8, 2024 21:48 Active

github-actions bot deployed to collection-api-dev November 8, 2024 21:48 Active

github-actions bot deployed to prospect-api-dev November 8, 2024 21:49 Active

github-actions bot deployed to collection-api-dev November 8, 2024 22:14 Active

github-actions bot deployed to collection-api-dev November 8, 2024 22:36 Active

jpetto force-pushed the MC-1588-add-otlp-to-collection-api branch from 691f76c to f469c75 Compare November 8, 2024 23:02

jpetto marked this pull request as ready for review November 8, 2024 23:03

jpetto marked this pull request as draft November 8, 2024 23:04

jpetto force-pushed the MC-1588-add-otlp-to-collection-api branch from 6746003 to 7963998 Compare November 12, 2024 17:13

bassrock reviewed Nov 12, 2024

View reviewed changes

bassrock previously approved these changes Nov 12, 2024

View reviewed changes

jpetto marked this pull request as ready for review November 12, 2024 17:50

katerinachinnappan reviewed Nov 13, 2024

View reviewed changes

README.md

To disable tracing on a service, simply toggle the `default` environment to "Off".

### Local Tracing

Copy link

Collaborator

katerinachinnappan Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

katerinachinnappan previously approved these changes Nov 13, 2024

View reviewed changes

jpetto added 5 commits November 13, 2024 10:41

feat: add tracing to collection API

585c9cb

- uses pocket's OTLP collector endpoint - sends traces AND logs to GCP for easy viewing

update sentry for corpus and prospect API

5a4ad15

reduce prisma logging

191e996

reduce logging level to 'info' in AWS

ecd5be0

update README

e914567

jpetto added 2 commits November 13, 2024 10:41

cleanup

693b2a4

small README updates

a86912a

jpetto dismissed stale reviews from katerinachinnappan and bassrock via a86912a November 13, 2024 16:42

jpetto force-pushed the MC-1588-add-otlp-to-collection-api branch from 8995adc to a86912a Compare November 13, 2024 16:42

bassrock approved these changes Nov 13, 2024

View reviewed changes

jpetto merged commit 1d0f1c4 into main Nov 13, 2024
49 checks passed

jpetto deleted the MC-1588-add-otlp-to-collection-api branch November 13, 2024 16:56

jpetto mentioned this pull request Nov 15, 2024

feat: add tracing to corpus API #244

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add tracing to collection API #236

feat: add tracing to collection API #236

jpetto commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

bassrock Nov 12, 2024

katerinachinnappan Nov 13, 2024

katerinachinnappan Nov 13, 2024

katerinachinnappan left a comment


		To disable tracing on a service, simply toggle the `default` environment to "Off".

		### Local Tracing

feat: add tracing to collection API #236

feat: add tracing to collection API #236

Conversation

jpetto commented Nov 8, 2024 • edited Loading

Goal

Implementation Decisions

Deployment steps

References

github-actions bot commented Nov 8, 2024 • edited Loading

Plan Result (prospect-translation-lambda-cdk-production)

⚠️ Errors

github-actions bot commented Nov 8, 2024 • edited Loading

Plan Result (corpus-scheduler-lambda-cdk-production)

⚠️ Errors

github-actions bot commented Nov 8, 2024 • edited Loading

Plan Result (prospect-api-cdk-production)

github-actions bot commented Nov 8, 2024 • edited Loading

Plan Result (collection-api-cdk-production)

⚠️ Resource Deletion will happen

bassrock Nov 12, 2024

Choose a reason for hiding this comment

katerinachinnappan Nov 13, 2024

Choose a reason for hiding this comment

katerinachinnappan Nov 13, 2024

Choose a reason for hiding this comment

katerinachinnappan left a comment

Choose a reason for hiding this comment

jpetto commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading

github-actions bot commented Nov 8, 2024 •

edited

Loading