Experimentation

Run A/B tests and holdout experiments with weighted variants and statistical significance.

Experiments let you test different email variants — subject lines, templates, timing — and measure which performs best. Molted Email supports A/B tests and holdout experiments with statistical significance testing.

Create an experiment

POST https://api.molted.email/v1/experiments

curl

curl -X POST https://api.molted.email/v1/experiments \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tenantId": "tenant_abc123",
    "name": "Welcome email subject test",
    "type": "ab",
    "journeyId": "JOURNEY_ID",
    "segmentId": "SEGMENT_ID",
    "variants": [
      {
        "name": "Control",
        "weight": 0.5,
        "templateVersionId": "tv_001",
        "isControl": true
      },
      {
        "name": "Casual subject",
        "weight": 0.5,
        "templateVersionId": "tv_001",
        "subjectOverride": "Hey! Welcome aboard"
      }
    ]
  }'

Field	Type	Required	Description
`tenantId`	`string`	Yes	Your tenant identifier.
`name`	`string`	Yes	Experiment name.
`type`	`string`	Yes	`ab` or `holdout`.
`journeyId`	`string`	No	Journey to integrate with.
`segmentId`	`string`	No	Segment to target.
`variants`	`array`	Yes	List of variants (see below).

Variant fields

Field	Type	Required	Description
`name`	`string`	Yes	Variant display name.
`weight`	`number`	Yes	Traffic allocation (0–1). All weights must sum to 1.
`templateVersionId`	`string`	Yes	Template version to use.
`subjectOverride`	`string`	No	Override subject line for this variant.
`timingDelayMinutes`	`number`	No	Delay before sending (for timing tests).
`isControl`	`boolean`	No	Mark as the control variant.
`isHoldout`	`boolean`	No	Holdout variant — no email is sent, used to measure lift.

List experiments

GET https://api.molted.email/v1/experiments

curl

curl "https://api.molted.email/v1/experiments?tenantId=tenant_abc123&journeyId=JOURNEY_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

Get experiment details

GET https://api.molted.email/v1/experiments/:id

Start an experiment

POST https://api.molted.email/v1/experiments/:id/start

curl

curl -X POST https://api.molted.email/v1/experiments/EXPERIMENT_ID/start \
  -H "Authorization: Bearer YOUR_API_KEY"

Stop an experiment

POST https://api.molted.email/v1/experiments/:id/stop

curl

curl -X POST https://api.molted.email/v1/experiments/EXPERIMENT_ID/stop \
  -H "Authorization: Bearer YOUR_API_KEY"

Get results

GET https://api.molted.email/v1/experiments/:id/results

curl

curl https://api.molted.email/v1/experiments/EXPERIMENT_ID/results \
  -H "Authorization: Bearer YOUR_API_KEY"

Response

{
  "experimentId": "EXPERIMENT_ID",
  "status": "completed",
  "variants": [
    {
      "name": "Control",
      "sent": 500,
      "conversions": 45,
      "conversionRate": 0.09,
      "isControl": true
    },
    {
      "name": "Casual subject",
      "sent": 500,
      "conversions": 62,
      "conversionRate": 0.124,
      "delta": 0.034,
      "pValue": 0.032,
      "significant": true
    }
  ]
}

Results include a pValue and significant flag (95% confidence level) for each non-control variant.

How assignment works

Contacts are assigned to variants deterministically based on a hash of the experiment ID and contact email. This ensures the same contact always sees the same variant, even across retries.

Holdout variants skip sending entirely. The decision is logged so you can measure the lift of sending vs. not sending.

Journey integration

Experiments can be linked to a journey step. When a journey reaches a send step with an experiment, the contact is assigned to a variant and the corresponding template/subject is used.

On this page