Skip to content

Commit

Permalink
Merge branch 'main' into docs-piping-dvc-data-to-external-destinations
Browse files Browse the repository at this point in the history
  • Loading branch information
andrewdmaclean authored Jul 2, 2024
2 parents a0c84fc + 964f238 commit c0f7719
Show file tree
Hide file tree
Showing 12 changed files with 78 additions and 35 deletions.
4 changes: 2 additions & 2 deletions docs/best-practices/engineering-led-experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ To complete this guide, you may want to review the following topics:

- [Creating and Managing Variations](/essentials/variables)
- [Random Distribution Targeting](/essentials/targeting)
- [Feature Experimentation](/extras/metrics/feature-experimentation)
- [Feature Experimentation](/essentials/feature-experimentation)
- [Creating a Metric](/extras/metrics/creating-and-managing-metrics) and [Metric Types](/extras/metrics/creating-and-managing-metrics#types)

## Metrics in DevCycle
Expand Down Expand Up @@ -116,7 +116,7 @@ Some of the simplest and most precise experiments are basic A/B tests comparing

For more complex features you may compare more variations with several combinations of variable values. You would then randomly distribute the variations according to the number of variations you are including in the experiment.

For more information about comparing multiple variations in an experiment, refer to [Feature Experimentation](/extras/metrics/feature-experimentation#comparing-multiple-variations).
For more information about comparing multiple variations in an experiment, refer to [Feature Experimentation](/essentials/feature-experimentation#comparing-multiple-variations).

### Choosing Metrics

Expand Down
2 changes: 1 addition & 1 deletion docs/best-practices/feature-grouping.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This guide describes how to manage large amounts of feature flags with DevCycle.

## Why Group Flags?

Feature Flags can accumulate very quickly. Without an organizational mechanism, the feature management dashboard would be extremely cluttered. Imagine developing a complex feature with various parts, such as the [Metrics and Experimentation feature in DevCycle](/extras/metrics/feature-experimentation). Each piece of functionality needs a flag: There’s a flag for the API, the UI, each type of experimentation, string flags for naming, and so on. It would be helpful to have the ability to find all these flags in the same place.
Feature Flags can accumulate very quickly. Without an organizational mechanism, the feature management dashboard would be extremely cluttered. Imagine developing a complex feature with various parts, such as the [Metrics and Experimentation feature in DevCycle](/essentials/feature-experimentation). Each piece of functionality needs a flag: There’s a flag for the API, the UI, each type of experimentation, string flags for naming, and so on. It would be helpful to have the ability to find all these flags in the same place.

Not only do we want flags for each part of a feature, but we also want the ability to turn off the entire feature, all at once, in case something goes wrong. Other Feature Flag management solutions require you to create complex feature flag dependencies to set up a “master switch” for a group of flags. In contrast, DevCycle allows you to easily group feature flags and disable them simultaneously with one switch.

Expand Down
2 changes: 1 addition & 1 deletion docs/best-practices/product-led-experimentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ To complete this guide, you may want to review the following topics:
- [DevCycle Feature Hierarchy](/introduction/core-concepts/feature-hierarchy)
- [Creating and Managing Variations](/essentials/variables)
- [Random Distribution Targeting](/essentials/targeting)
- [Feature Experimentation](/extras/metrics/feature-experimentation)
- [Feature Experimentation](/essentials/feature-experimentation)
- [Creating a Metric](/extras/metrics/creating-and-managing-metrics) and [Metric Types](/extras/metrics/creating-and-managing-metrics#types)

## Conducting Experiments: A Collaborative Approach
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,8 @@
---
title: Feature Experimentation
sidebar_position: 3
sidebar_position: 10
---

:::info
Experimentation relies on custom events.

Experimentation is available to all customers on any plan. However, to perform experiments, events must be sent to DevCycle to calculate metrics. These events are added to your existing plan. To learn more, read about our pricing, or contact us.
:::

## Overview

At DevCycle we believe that experimentation should be a part of the natural lifecycle of all features. So no matter the [feature type](/essentials/features) selected, can be experimented on. Experiments can be as simple as comparing any target audiences against a metric, or can be fully [randomized](/essentials/targeting#serving-a-random-variation-experimentation--random-distribution) A/B tests using statistical methodologies.
Expand Down Expand Up @@ -49,6 +43,12 @@ To set this up, create a targeting rule in Production that delivers to All Users

## Adding Metrics to Your Feature

:::info
Experimentation relies on custom events.

Experimentation is available to all customers on any plan. However, to perform experiments, events must be sent to DevCycle to calculate metrics. These events are added to your existing plan. To learn more, read about our pricing, or contact us.
:::

Now that you have two segments receiving different experiences, the only other thing you need to run an experiment is a metric to evaluate the comparative performance of those experiences.

To add a metric to your feature, click “Comparative Analysis” under the “Data & Results” section on the sidebar of the feature editing page. Click the “Choose a Metric” dropdown. This will bring up the option to add a metric that has already been created in the project or to create a new one.
Expand Down
6 changes: 3 additions & 3 deletions docs/extras/metrics/creating-and-managing-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ To run a test, the following fields must be set:

**Feature** - This is the specific feature this Metric should be applied to. Any event that has been sent since the creation of this Metric from a user receiving any variation of this feature will be part of this Metric. In the event that an error is shown, this means the event has not been seen from this feature yet.

**Control** - After selecting a Feature, a "control" variation must be selected. This is what will be used to show a comparative analysis against all other variations in a feature. Typically, an "off" or "Baseline" variation would act as the control. For more information on this, please refer to the [Feature Experimentation documentation](/extras/metrics/feature-experimentation).
**Control** - After selecting a Feature, a "control" variation must be selected. This is what will be used to show a comparative analysis against all other variations in a feature. Typically, an "off" or "Baseline" variation would act as the control. For more information on this, please refer to the [Feature Experimentation documentation](/essentials/feature-experimentation).

**Date Range** - This range will default from the moment the Metric (or feature) was created, to now within 30 days. If 30 days have passed since the creation of the Metric or feature, the date range must be a 30 day range.

Expand Down Expand Up @@ -132,7 +132,7 @@ Here are some steps you can follow to track metrics within a feature:

While the setup has some default values, the Metric requires the following fields to be filled:

- **Control** - This is what will be used to show a comparative analysis against all other variations in a feature. Typically, an "off" or "Baseline" variation would act as the control. For more information on this, please refer to the [Feature Experimentation documentation](/extras/metrics/feature-experimentation).
- **Control** - This is what will be used to show a comparative analysis against all other variations in a feature. Typically, an "off" or "Baseline" variation would act as the control. For more information on this, please refer to the [Feature Experimentation documentation](/essentials/feature-experimentation).

- **Date Range** - This range will default from the moment the Metric (or feature) was created, to now within 30 days. If 30 days have passed since the creation of the Metric or feature, the date range must be a 30 day range.

Expand All @@ -158,4 +158,4 @@ To read more on the queries behind the Metrics, see [How Metrics Are Calculated]

With Metrics on a feature, experimentation can be easily executed on any Feature. At DevCycle we believe that experimentation should be a part of the natural lifecycle of all features. So no matter the [feature type](/essentials/features) selected, experimentation will always be available for use.

To learn more on how to run experiments with DevCycle, read [Feature Experimentation](/extras/metrics/feature-experimentation)
To learn more on how to run experiments with DevCycle, read [Feature Experimentation](/essentials/feature-experimentation)
2 changes: 1 addition & 1 deletion docs/extras/web-debugger.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ Once set, you can now view and modify your Overrides for any Feature in your pro
In the "Live Configuration" section, you'll find a list of the Features and Variations that you are currently being served by DevCycle on this page.
Each row can be clicked to expand and see the set of Variables and their values which are associated with that Feature and Variation.

At the top of this section you can also see the current user data that you've been identified with on this page, and modify any property.
At the top of this section you can also see the current user data that you've been identified with on this page, and modify User ID or Email and/or any custom property.
Applying changes to that data will trigger the SDK's `identify` function and trigger a new configuration to be retrieved from DevCycle.

### Events
Expand Down
9 changes: 5 additions & 4 deletions docs/sdk/server-side-sdks/go/go-gettingstarted.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ initialize for any reason- it'll return as an error here.

## Async Initialization

Additionally, local bucketing mode supports an optional `OnInitializedChannel` parameter which will tell the sdk to run the initialization
process in a separate go routine. When the channel receives a message, you will know the initialization process is complete.
Additionally, local bucketing mode supports an optional `ClientEventHandler` parameter which will tell the sdk to run the initialization
process in a separate go routine. This can be useful if you want to wait for the client to be fully initialized before proceeding.

```go
onInitializedChannel := make(chan bool)
onInitializedChannel := make(chan api.ClientEvent)
options := devcycle.Options{
OnInitializedChannel: onInitializedChannel,
ClientEventHandler: onInitializedChannel,
// other options omitted for this example
}

Expand All @@ -73,3 +73,4 @@ log.Println("DevCycle client not guaranteed to be initialized yet")
<-onInitializedChannel
log.Println("Devcycle client initialized")
```

21 changes: 21 additions & 0 deletions docs/sdk/server-side-sdks/go/go-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,24 @@ making bucketing decisions.
In this example, the `Variable` call would make an API request and send along the user data specified by the call.
That data would be used in combination with EdgeDB data to make correct bucketing decisions before returning the
variable's value.


## Enabling Beta Realtime Updates
:::warning
This feature is in beta, and may not function as expected. Please ensure that you have the latest version of the SDK.
:::

This functionality will reduce the number of polling requests that are made to the DevCycle Config CDN, and instead will
use a long-lived HTTP connection (Server Sent Events) to receive updates when there is a new config available.
This reduces outbound network traffic, as well as optimizes the SDK for efficiency.

To enable Beta Realtime Updates, pass in the `EnableBetaRealtimeUpdates` option to the SDK initialization:
```go
options := devcycle.Options{
EnableBetaRealtimeUpdates: true,
// other options omitted for this example
}

devcycleClient, err := devcycle.NewClient(sdkKey, &options)
```

3 changes: 2 additions & 1 deletion docs/sdk/server-side-sdks/node/node-gettingstarted.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ const devcycleClient = await DevCycle.initializeDevCycle(
```

| DevCycle Option | Type | Description |
| ---------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|------------------------------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| logger | DevCycleLogger | Logger override to replace default logger |
| logLevel | String | Set log level of the default logger. Options are: `debug`, `info`, `warn`, `error`. Defaults to `info`. |
| enableCloudBucketing | Boolean | Switches the SDK to use Cloud Bucketing (via the DevCycle Bucketing API) instead of Local Bucketing. |
Expand All @@ -65,3 +65,4 @@ const devcycleClient = await DevCycle.initializeDevCycle(
| flushEventQueueSize | Number | Controls the maximum size the event queue can grow to until a flush is forced. Defaults to `1000`. |
| maxEventQueueSize | Number | Controls the maximum size the event queue can grow to until events are dropped. Defaults to `2000`. |
| apiProxyURL | String | Allows the SDK to communicate with a proxy of DevCycle bucketing API / client SDK API. |
| enableBetaRealtimeUpdates | Boolean | Enables the usage of Beta Realtime Updates for DevCycle. This feature is currently in beta. |
20 changes: 20 additions & 0 deletions docs/sdk/server-side-sdks/node/node-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,26 @@ This will send a request to our EdgeDB API to save the custom data under the use

In the example, Email and Country are associated to the user `test_user`. In your next variable call for the same `user_id`, you may omit any of the data you've sent already as it will be pulled from the EdgeDB storage when segmenting to experiments and features.

## Enabling Beta Realtime Updates
:::warning
This feature is in beta, and may not function as expected. Please ensure that you have the latest version of the SDK.
:::

This functionality will reduce the number of polling requests that are made to the DevCycle Config CDN, and instead will
use a long-lived HTTP connection (Server Sent Events) to receive updates when there is a new config available.
This reduces outbound network traffic, as well as optimizes the SDK for efficiency.

To enable Beta Realtime Updates, pass in the `enableBetaRealtimeUpdates` option to the SDK initialization:
```javascript
const devcycleClient = DevCycle.initializeDevCycle(
'<DEVCYCLE_SERVER_SDK_KEY>',
{
enableBetaRealtimeUpdates: true,
},
)
```


## Close Client

If you need to close the DevCycleClient object to stop all open connections and timers, call `devcycleClient.close()`.
Expand Down
2 changes: 1 addition & 1 deletion docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ const YouTubeTransformer = {
* Pinned version of the CLI to use for docs
* When bumping the version, add any new commands to the documents array
*/
const DVC_CLI_VERSION = 'v5.14.12' // auto updated by dvc cli release workflow
const DVC_CLI_VERSION = 'v5.14.13' // auto updated by dvc cli release workflow

const VSCODE_EXTENSION_VERSION = 'v1.4.10' // auto updated by extension release workflow

Expand Down
28 changes: 14 additions & 14 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -5530,11 +5530,11 @@ __metadata:
linkType: hard

"braces@npm:^3.0.2, braces@npm:~3.0.2":
version: 3.0.2
resolution: "braces@npm:3.0.2"
version: 3.0.3
resolution: "braces@npm:3.0.3"
dependencies:
fill-range: "npm:^7.0.1"
checksum: 10/966b1fb48d193b9d155f810e5efd1790962f2c4e0829f8440b8ad236ba009222c501f70185ef732fef17a4c490bb33a03b90dab0631feafbdf447da91e8165b1
fill-range: "npm:^7.1.1"
checksum: 10/fad11a0d4697a27162840b02b1fad249c1683cbc510cd5bf1a471f2f8085c046d41094308c577a50a03a579dd99d5a6b3724c4b5e8b14df2c4443844cfcda2c6
languageName: node
linkType: hard

Expand Down Expand Up @@ -7976,12 +7976,12 @@ __metadata:
languageName: node
linkType: hard

"fill-range@npm:^7.0.1":
version: 7.0.1
resolution: "fill-range@npm:7.0.1"
"fill-range@npm:^7.1.1":
version: 7.1.1
resolution: "fill-range@npm:7.1.1"
dependencies:
to-regex-range: "npm:^5.0.1"
checksum: 10/e260f7592fd196b4421504d3597cc76f4a1ca7a9488260d533b611fc3cefd61e9a9be1417cb82d3b01ad9f9c0ff2dbf258e1026d2445e26b0cf5148ff4250429
checksum: 10/a7095cb39e5bc32fada2aa7c7249d3f6b01bd1ce461a61b0adabacccabd9198500c6fb1f68a7c851a657e273fce2233ba869638897f3d7ed2e87a2d89b4436ea
languageName: node
linkType: hard

Expand Down Expand Up @@ -16132,8 +16132,8 @@ __metadata:
linkType: hard

"ws@npm:^7.3.1":
version: 7.5.9
resolution: "ws@npm:7.5.9"
version: 7.5.10
resolution: "ws@npm:7.5.10"
peerDependencies:
bufferutil: ^4.0.1
utf-8-validate: ^5.0.2
Expand All @@ -16142,13 +16142,13 @@ __metadata:
optional: true
utf-8-validate:
optional: true
checksum: 10/171e35012934bd8788150a7f46f963e50bac43a4dc524ee714c20f258693ac4d3ba2abadb00838fdac42a47af9e958c7ae7e6f4bc56db047ba897b8a2268cf7c
checksum: 10/9c796b84ba80ffc2c2adcdfc9c8e9a219ba99caa435c9a8d45f9ac593bba325563b3f83edc5eb067cc6d21b9a6bf2c930adf76dd40af5f58a5ca6859e81858f0
languageName: node
linkType: hard

"ws@npm:^8.13.0":
version: 8.17.0
resolution: "ws@npm:8.17.0"
version: 8.17.1
resolution: "ws@npm:8.17.1"
peerDependencies:
bufferutil: ^4.0.1
utf-8-validate: ">=5.0.2"
Expand All @@ -16157,7 +16157,7 @@ __metadata:
optional: true
utf-8-validate:
optional: true
checksum: 10/5e1dcb0ae70c6e2f158f5b446e0a72a2cd335b07aba73ee1872e9bae1285382286a10e53ed479db21bdd690a5dfd05641a768611ebb236253c62fefa43ef58b4
checksum: 10/4264ae92c0b3e59c7e309001e93079b26937aab181835fb7af79f906b22cd33b6196d96556dafb4e985742dd401e99139572242e9847661fdbc96556b9e6902d
languageName: node
linkType: hard

Expand Down

0 comments on commit c0f7719

Please sign in to comment.