tea-tasting design questions and ideas

[e10v]

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test, Z-test, and Bootstrap out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Fieller's confidence interval for percent change.
• Sample ratio mismatch check.
• Power analysis.
• A/A tests.

Currently, tea-tasting is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).

Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach.

In this discussion, I welcome any questions and ideas about the package features and API design.

---

This discussion was created from the release tea-tasting 0.0.1.