Experimentation

Experiments let you test different email variants -- subject lines, templates, timing -- and measure which performs best. Molted Email supports A/B tests and holdout experiments with statistical significance testing.

On api.molted.email, pass tenantId (in the body for writes, in the query string for reads); mailbox.molted.email and the CLI fill it in. A mailbox-scoped key is refused on writes and on results. There are no MCP tools for experiments yet: use the CLI or the API.

Create an experiment

POST https://mailbox.molted.email/v1/experiments

curl -X POST https://mailbox.molted.email/v1/experiments \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Welcome email subject test",
    "type": "ab",
    "journeyId": "JOURNEY_ID",
    "journeyStepId": "JOURNEY_STEP_ID",
    "segmentId": "SEGMENT_ID",
    "variants": [
      {
        "id": "v1",
        "name": "Control",
        "weight": 50,
        "templateVersionId": "tv_001",
        "isControl": true
      },
      {
        "id": "v2",
        "name": "Casual subject",
        "weight": 50,
        "templateVersionId": "tv_001",
        "subjectOverride": "Hey! Welcome aboard"
      }
    ]
  }'

molted experiments create --name "Welcome email subject test" --type ab \
  --journey JOURNEY_ID --journey-step JOURNEY_STEP_ID --segment SEGMENT_ID \
  --variants '[
    {"id":"v1","name":"Control","weight":50,"templateVersionId":"tv_001","isControl":true,"isHoldout":false},
    {"id":"v2","name":"Casual subject","weight":50,"templateVersionId":"tv_001","subjectOverride":"Hey! Welcome aboard","isControl":false,"isHoldout":false}
  ]'
# --variants also accepts @path/to/variants.json

journeyId, journeyStepId and segmentId must belong to your own workspace, and a journey that was deleted does not count; when both journeyId and journeyStepId are given, the step must belong to that journey. Otherwise the API returns 404 (journey_not_found, journey_step_not_found or segment_not_found), the same answer as for an id that does not exist. The id fields must be UUIDs (anything else is a 400 invalid_id). The response is the experiment as a raw row: { id, tenant_id, journey_id, journey_step_id, name, type, status, segment_id, variants, created_at, updated_at }, with status draft, running or completed. The same shape comes back from list, get, start and stop.

Variant fields

Only the CLI validates variants (unique ids, positive weights). The API stores the array as given, so check it yourself when calling the API directly.

templateId is not a variant field: the API silently ignores it. Use templateVersionId (from a template's versions, see Templates).

Caveat: a variant's templateVersionId is stored and shown in results, but the send is still rendered from the journey step's own base template, not re-rendered from the variant's template version (wiring that up is tracked in #1490). subjectOverride DOES take effect today. Until #1490 ships, use subjectOverride for the actual content difference between variants.

List experiments

GET https://mailbox.molted.email/v1/experiments

curl "https://mailbox.molted.email/v1/experiments?journeyId=JOURNEY_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

molted experiments list --journey JOURNEY_ID

Get experiment details

GET https://mailbox.molted.email/v1/experiments/:id

curl "https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

molted experiments get EXPERIMENT_ID

Start an experiment

POST https://mailbox.molted.email/v1/experiments/:id/start

curl -X POST https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/start \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

molted experiments start EXPERIMENT_ID

Only works from draft; it sets status to running. On an unknown experiment, or one not in draft, the API returns an empty 2xx body rather than an error (the CLI turns that into an error and exit 1).

Stop an experiment

POST https://mailbox.molted.email/v1/experiments/:id/stop

curl -X POST https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/stop \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

molted experiments stop EXPERIMENT_ID

Only works from running; it sets status to completed. Like start, a wrong state or unknown id returns an empty 2xx body.

Get results

GET https://mailbox.molted.email/v1/experiments/:id/results

curl "https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/results" \
  -H "Authorization: Bearer YOUR_API_KEY"

molted experiments results EXPERIMENT_ID

Response

An array with one entry per variant:

[
  {
    "variantId": "v1",
    "variantName": "Control",
    "totalAssigned": 510,
    "totalSent": 500,
    "totalDelivered": 492,
    "totalConverted": 45,
    "conversionRate": 0.09,
    "isSignificant": false,
    "pValue": null,
    "confidenceInterval": null
  },
  {
    "variantId": "v2",
    "variantName": "Casual subject",
    "totalAssigned": 505,
    "totalSent": 500,
    "totalDelivered": 495,
    "totalConverted": 62,
    "conversionRate": 0.124,
    "isSignificant": true,
    "pValue": 0.032,
    "confidenceInterval": [0.002, 0.066]
  }
]

conversionRate is totalConverted / totalSent. Each non-control variant is compared with the control using a two-proportion z-test: isSignificant means p < 0.05, and confidenceInterval is the 95% interval. The control keeps pValue: null. An unknown experiment id returns an empty 200.

Conversions are outcomes you record (POST /v1/outcomes/ingest, molted outcomes ingest, MCP record_outcome) that get attributed to a variant's send.

Results in the portal

A person sees an experiment's results in the portal on the page of the journey it runs in: /app/journeys/<id>, in an Experiment card under the journey's steps and runs. There is no separate Experiments page.

The card shows the experiment's name and, for each variant, how many contacts were assigned, how many emails were sent, and how many converted, with the conversion rate and a bar scaled to the best variant. Once anything has been sent, the variant with the highest conversion rate carries a Leading badge. That badge compares plain conversion rates; it is not a significance test. For pValue, isSignificant and the confidence interval, read the results from the API.

The page shows one experiment per journey (the first it finds), and it only reads: creating, starting and stopping an experiment stay with your agents.

How assignment works

Contacts are assigned to variants deterministically based on a SHA-256 hash of the experiment ID and contact email, and the first assignment is stored, so the same contact always sees the same variant, even across retries.

Holdout variants skip sending entirely. The decision is logged, but since nothing is sent, a holdout has no send for an outcome to be attributed to: its totalSent and conversionRate stay 0 in the results, so the results can't show holdout lift today.

Journey integration

Experiments can be linked to a journey step via journeyStepId (see the field table above -- required in practice, not just in name). When a journey reaches a send step whose id matches a running experiment's journeyStepId, the contact is assigned to a variant. The variant's subjectOverride is applied to that send; templateVersionId is recorded but does not currently change what's rendered (see the caveat above, #1490).