Experimentation
A/B Testing on Live Journeys
Attach an experiment to any journey step. We handle variant assignment (deterministic, SHA-256 based), decision logging, and significance testing. You just pick what to test.
How it works
Create an experiment
POST to /v1/experiments with a journey step ID and weighted variants. Each variant can override the subject line, swap in a different template version, or act as a holdout group.
Start it
POST to /v1/experiments/:id/start. Moves the experiment from draft to running. Only running experiments touch live sends, so you can set everything up safely first.
Contacts get assigned deterministically
When the journey step fires, each contact is hashed (SHA-256) into a variant bucket. Same contact, same variant, every time. Holdout contacts skip the send entirely.
Overrides kick in
Subject line and template overrides from the assigned variant are applied before sending. Every decision is logged for attribution.
Outcomes traced back
When business events arrive (purchases, signups, activations), they trace back through experiment decision logs. Four attribution models are supported.
You get significance numbers
GET /v1/experiments/:id/results returns per-variant conversion rates, a two-proportion z-test, p-values, and 95% confidence intervals. No spreadsheet required.
Create an experiment
POST /v1/experiments
{
"tenantId": "your-tenant-id",
"journeyId": "welcome-series",
"journeyStepId": "step-2-nudge",
"name": "Subject line test",
"type": "ab",
"variants": [
{
"id": "control",
"name": "Control",
"weight": 50,
"isControl": true,
"isHoldout": false,
"subjectOverride": null,
"templateVersionId": null,
"timingDelayMinutes": null
},
{
"id": "variant-b",
"name": "Urgency subject",
"weight": 50,
"isControl": false,
"isHoldout": false,
"subjectOverride": "Don't miss out - your trial ends tomorrow",
"templateVersionId": null,
"timingDelayMinutes": null
}
]
}What you can test
Subject lines
The lowest-friction test you can run. Swap subject copy across variants and measure open and conversion rate differences.
Template versions
Different designs, different layouts, different copy. Point each variant at a separate template version and let the data decide.
Holdout groups
Would this email even matter if you didn't send it? Withhold from a control group and compare outcomes to find out.
Significance, not guesswork
Results use a two-proportion z-test at 95% confidence. Once enough sends accumulate, you get a clear signal: p-value, confidence interval, and a flag telling you whether the difference is real.
Variant Sent Converted Rate p-value Control 1,204 83 6.89% - Urgency 1,198 112 9.35% 0.0241 *
* Statistically significant at p < 0.05
Run your first experiment
One POST creates the experiment, another starts it. Significance tracking happens on its own. Ship a test in under five minutes.