Experimentation

A/B Testing for Email Journeys

Run experiments directly on journey steps. Molted Email handles variant assignment, decision logging, and statistical significance — you just pick what to test.

How it works

1

Create an experiment

POST to /v1/experiments with a journey step ID and weighted variants. Each variant can override the subject line, use a different template version, or act as a holdout group.

2

Start the experiment

POST to /v1/experiments/:id/start to move it from draft to running. Only running experiments affect live sends.

3

Automatic variant assignment

When the journey step executes, the system assigns each contact deterministically using SHA-256 hashing. The same contact always gets the same variant. Holdout contacts skip the send entirely.

4

Variant overrides applied

Subject line and template overrides from the assigned variant are applied automatically. Every decision is logged for attribution.

5

Attribution tracking

When outcome events arrive (purchases, signups, activations), they are traced back through experiment decision logs. Four models are supported: last-touch, first-touch, linear, and time-decay.

6

Results and significance

GET /v1/experiments/:id/results returns per-variant stats and runs a two-proportion z-test — p-value, significance flag, and 95% confidence interval.

Create an experiment

POST /v1/experiments

{
  "tenantId": "your-tenant-id",
  "journeyId": "welcome-series",
  "journeyStepId": "step-2-nudge",
  "name": "Subject line test",
  "type": "ab",
  "variants": [
    {
      "id": "control",
      "name": "Control",
      "weight": 50,
      "isControl": true,
      "isHoldout": false,
      "subjectOverride": null,
      "templateVersionId": null,
      "timingDelayMinutes": null
    },
    {
      "id": "variant-b",
      "name": "Urgency subject",
      "weight": 50,
      "isControl": false,
      "isHoldout": false,
      "subjectOverride": "Don't miss out — your trial ends tomorrow",
      "templateVersionId": null,
      "timingDelayMinutes": null
    }
  ]
}

What you can test

Subject lines

Test different subject copy to see which drives higher open and conversion rates.

Template versions

Test entirely different email designs, layouts, or copy against each other.

Holdout groups

Measure incremental impact by withholding email from a control group and comparing outcomes.

Automatic significance tracking

Results are computed with a two-proportion z-test at a 95% confidence level. Once enough data accumulates, you get a clear significance signal with p-values and confidence intervals.

Variant        Sent     Converted   Rate     p-value
Control        1,204    83          6.89%    —
Urgency        1,198    112         9.35%    0.0241  *

* Statistically significant at p < 0.05

Start experimenting

Create your first A/B test in minutes. Molted Email handles assignment, tracking, and significance — you just pick what to test.

Get started free