feat(flags): Add integration for custom tracking of flag evaluations

[aliu39]

Follow-up to

• Add LaunchDarkly and OpenFeature integration #3648

1. Adds an integration for buffering feature flags manually with an API (add_feature_flag), and automatically capturing them on error events.

2. Adds improved test coverage to launchdarkly and openfeature by testing end to end with capture_events fixture, rather than asserting the flags are stored to scope.

[codecov]

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
15097	4	15093	4390

View the top 3 failed tests by shortest run time

tests.integrations.featureflags.test_featureflags test_featureflags_integration_asyncio

Stack Traces | 0.067s run time

.../integrations/featureflags/test_featureflags.py:103: in test_featureflags_integration_asyncio
    asyncio.run(runner())
E   AttributeError: module 'asyncio' has no attribute 'run'

tests.integrations.featureflags.test_featureflags test_featureflags_integration_asyncio

Stack Traces | 0.068s run time

.../integrations/featureflags/test_featureflags.py:103: in test_featureflags_integration_asyncio
    asyncio.run(runner())
E   AttributeError: module 'asyncio' has no attribute 'run'

tests.integrations.launchdarkly.test_launchdarkly test_launchdarkly_integration_threaded

Stack Traces | 0.102s run time

.../integrations/launchdarkly/test_launchdarkly.py:91: in test_launchdarkly_integration_threaded
    assert len(events) == 3
E   AssertionError: assert 4 == 3
E    +  where 4 = len([{'breadcrumbs': {'values': []}, 'contexts': {'flags': {'values': [{'flag': 'hello', 'result': True}, {'flag': 'world', 'result': False}]}, 'runtime': {'build': '3.13.1 (main, Dec  4 2024, 06:21:55) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.13.1'}, 'trace': {'parent_span_id': None, 'span_id': 'ba4ad5d3dddfa4d7', 'trace_id': 'b86666664d6b43c1bc85d8ef609d935b'}}, 'environment': 'production', 'event_id': '8c36fb14cdc24ff288468a25754dd532', ...}, {'breadcrumbs': {'values': []}, 'contexts': {'flags': {'values': [{'flag': 'hello', 'result': True}, {'flag': 'other', 'result': False}]}, 'runtime': {'build': '3.13.1 (main, Dec  4 2024, 06:21:55) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.13.1'}, 'trace': {'parent_span_id': None, 'span_id': 'ba4ad5d3dddfa4d7', 'trace_id': 'b86666664d6b43c1bc85d8ef609d935b'}}, 'environment': 'production', 'event_id': 'ca0792695a3f48c2a96dd00cab882c33', ...}, {'breadcrumbs': {'values': []}, 'contexts': {'runtime': {'build': '3.13.1 (main, Dec  4 2024, 06:21:55) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.13.1'}, 'trace': {'parent_span_id': None, 'span_id': '923e0ac338cd7129', 'trace_id': '5cb46fe1165a4118be8eaa9e3515ad0a'}}, 'environment': 'production', 'event_id': '7efc4ce3d76d442f95698cca800b0ad2', ...}, {'breadcrumbs': {'values': []}, 'contexts': {'flags': {'values': [{'flag': 'hello', 'result': True}]}, 'runtime': {'build': '3.13.1 (main, Dec  4 2024, 06:21:55) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.13.1'}, 'trace': {'parent_span_id': None, 'span_id': 'ba4ad5d3dddfa4d7', 'trace_id': 'b86666664d6b43c1bc85d8ef609d935b'}}, 'environment': 'production', 'event_id': '3e76e036f5d241ec966b26955d8b693f', ...}])

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

[aliu39]

Note on this fixture: I found using the reset_integrations fixture in conftest.py caused issues with the default dedupe integration. We just want the integration being tested to be setup at the start of every new test.

Without using this, the error processor isn't added correctly. This isn't a concern for real use, since user apps only call init once

[antonpirker]

Looks OK

[antonpirker]

We also need to add docs for this, can you prepare a PR in the docs repo?

[aliu39]

Sounds good! I've opened getsentry/sentry-docs#12152, will do a last check in the morning to make sure it's up to date!

[aliu39]

Do you know anything about the failing ubuntu tests @antonpirker ?

[antonpirker]

Do you know anything about the failing ubuntu tests @antonpirker ?

In Python 3.6 there is probably no asyncio.run().

Can you fix this somehow, or skip this test in Python 3.6 (then this integration will only be supported form Python 3.7 and up)