Experimentation

What it does

Experimentation is an add-on eligible from W4. It brings A/B testing into the same data model as product work, marketing and the ROI Ledger, so a result is linked to the feature, issue and flag it affects.

Design

Experiment design captures a hypothesis, a primary metric from the semantic layer, guardrail metrics, the audience, variants and allocation, and planned duration. A sample size and duration calculator uses the metric's baseline, the minimum detectable effect, power and significance, and warns when traffic is too low. The primary metric and analysis plan are frozen at start. Changes after start create a new analysis version flagged as post-hoc.

Experiment types cover product UI and features, messaging (linking native Marketing A/B tests as records) and pricing. Pricing and eligibility experiments need approval and a verified consumer-law fact before they start. An AI review assist flags likely dark patterns and misleading price displays for human review. Experiments involving children's data are blocked unless verifiable parental consent flows exist (verify DPDP).

Run

Assignment uses deterministic bucketing by subject key through Feature Flags SDKs or a server-side Action, with exposure logging. Subjects are pseudonymous keys. Experiment data does not store party IDs unless the tenant links them under a recorded purpose. Consent-aware assignment skips subjects without the required tracking consent.

Analyse and decide

Analysis is fixed-horizon frequentist with confidence intervals, with a sequential testing option for early stopping and a sample ratio mismatch check. An SRM-flagged experiment cannot be decided until someone acknowledges it with a reason. Guardrails raise a stop proposal to the owner when thresholds are crossed.

The decision record of winner, no difference or stop, with rationale, is R3 and only a person can make it. Ship the winner proposes a flag rollout and closes linked issues or features.

GA additions

GA plans include variance reduction, long-term holdouts aligned with ROI Ledger holdouts for agent attribution, mutually exclusive layers with an overlap calendar, a learnings library published to Knowledge, a Bayesian analysis option and an analyst agent that proposes designs and drafts summaries but never decides.