# Experimentation

Source: https://molted.email/docs/experimentation

Run A/B tests and holdout experiments with weighted variants and statistical significance.

Experiments let you test different email variants -- subject lines, templates, timing -- and measure which performs best. Molted Email supports A/B tests and holdout experiments with statistical significance testing.

On `api.molted.email`, pass `tenantId` (in the body for writes, in the query string for reads); `mailbox.molted.email` and the CLI fill it in. A mailbox-scoped key is refused on writes and on results. There are no MCP tools for experiments yet: use the CLI or the API.

## Create an experiment

```
POST https://mailbox.molted.email/v1/experiments
```

```bash title="curl"
curl -X POST https://mailbox.molted.email/v1/experiments \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Welcome email subject test",
    "type": "ab",
    "journeyId": "JOURNEY_ID",
    "journeyStepId": "JOURNEY_STEP_ID",
    "segmentId": "SEGMENT_ID",
    "variants": [
      {
        "id": "v1",
        "name": "Control",
        "weight": 50,
        "templateVersionId": "tv_001",
        "isControl": true
      },
      {
        "id": "v2",
        "name": "Casual subject",
        "weight": 50,
        "templateVersionId": "tv_001",
        "subjectOverride": "Hey! Welcome aboard"
      }
    ]
  }'
```

```bash title="CLI"
molted experiments create --name "Welcome email subject test" --type ab \
  --journey JOURNEY_ID --journey-step JOURNEY_STEP_ID --segment SEGMENT_ID \
  --variants '[
    {"id":"v1","name":"Control","weight":50,"templateVersionId":"tv_001","isControl":true,"isHoldout":false},
    {"id":"v2","name":"Casual subject","weight":50,"templateVersionId":"tv_001","subjectOverride":"Hey! Welcome aboard","isControl":false,"isHoldout":false}
  ]'
# --variants also accepts @path/to/variants.json
```

| Field | Type | Required | Description |
| - | - | - | - |
| `tenantId` | `string` | No | Only on `api.molted.email`, where it is required; `mailbox.molted.email` fills it in from your key. |
| `name` | `string` | Yes | Experiment name. |
| `type` | `string` | No | `ab` (default) or `holdout`. A label only: what makes a variant a holdout is its own `isHoldout`. |
| `journeyId` | `string` (UUID) | No | Journey to integrate with. |
| `journeyStepId` | `string` (UUID) | No (but required in practice) | The journey's `send` step this experiment runs on. The API accepts the experiment without it, but a journey step only picks up a `running` experiment whose `journeyStepId` matches that step's own id exactly -- an experiment created without one never actually runs. |
| `segmentId` | `string` (UUID) | No | Segment to target. A contact outside the segment gets the step's normal send with no variant assignment. A contact with no contact record skips the segment check. |
| `variants` | `array` | Yes | List of variants (see below). |

`journeyId`, `journeyStepId` and `segmentId` must belong to your own workspace, and a journey that was deleted does not count; when both `journeyId` and `journeyStepId` are given, the step must belong to that journey. Otherwise the API returns `404` (`journey_not_found`, `journey_step_not_found` or `segment_not_found`), the same answer as for an id that does not exist. The id fields must be UUIDs (anything else is a `400 invalid_id`). The response is the experiment as a raw row: `{ id, tenant_id, journey_id, journey_step_id, name, type, status, segment_id, variants, created_at, updated_at }`, with `status` `draft`, `running` or `completed`. The same shape comes back from list, get, start and stop.

### Variant fields

| Field | Type | Required | Description |
| - | - | - | - |
| `id` | `string` | Yes | Variant identifier, unique within the experiment (e.g. `v1`). |
| `name` | `string` | Yes | Variant display name. |
| `weight` | `number` | Yes | Relative traffic share, a positive number. Weights are normalized automatically, so they don't need to sum to 1 or 100: `50`/`50` behaves the same as `0.5`/`0.5`. |
| `templateVersionId` | `string \| null` | Yes | Template version to use, or `null` for a holdout. |
| `subjectOverride` | `string \| null` | No | Override subject line for this variant. |
| `timingDelayMinutes` | `number \| null` | No | Stored, but not applied today: it doesn't delay anything. |
| `isControl` | `boolean` | No | Mark as the control variant. Significance is only computed when one variant is the control. |
| `isHoldout` | `boolean` | No | Holdout variant: no email is sent. |

Only the CLI validates variants (unique ids, positive weights). The API stores the array as given, so check it yourself when calling the API directly.

`templateId` is not a variant field: the API silently ignores it. Use `templateVersionId` (from a template's `versions`, see [Templates](https://molted.email/docs/templates.md)).

Caveat: a variant's `templateVersionId` is stored and shown in results, but the send is still rendered from the journey step's own base template, not re-rendered from the variant's template version (wiring that up is tracked in #1490). `subjectOverride` DOES take effect today. Until #1490 ships, use `subjectOverride` for the actual content difference between variants.

## List experiments

```
GET https://mailbox.molted.email/v1/experiments
```

```bash title="curl"
curl "https://mailbox.molted.email/v1/experiments?journeyId=JOURNEY_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```bash title="CLI"
molted experiments list --journey JOURNEY_ID
```

## Get experiment details

```
GET https://mailbox.molted.email/v1/experiments/:id
```

```bash title="curl"
curl "https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```bash title="CLI"
molted experiments get EXPERIMENT_ID
```

## Start an experiment

```
POST https://mailbox.molted.email/v1/experiments/:id/start
```

```bash title="curl"
curl -X POST https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/start \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
```

```bash title="CLI"
molted experiments start EXPERIMENT_ID
```

Only works from `draft`; it sets `status` to `running`. On an unknown experiment, or one not in `draft`, the API returns an empty `2xx` body rather than an error (the CLI turns that into an error and exit 1).

## Stop an experiment

```
POST https://mailbox.molted.email/v1/experiments/:id/stop
```

```bash title="curl"
curl -X POST https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/stop \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
```

```bash title="CLI"
molted experiments stop EXPERIMENT_ID
```

Only works from `running`; it sets `status` to `completed`. Like start, a wrong state or unknown id returns an empty `2xx` body.

## Get results

```
GET https://mailbox.molted.email/v1/experiments/:id/results
```

```bash title="curl"
curl "https://mailbox.molted.email/v1/experiments/EXPERIMENT_ID/results" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

```bash title="CLI"
molted experiments results EXPERIMENT_ID
```

### Response

An array with one entry per variant:

```json title="Response"
[
  {
    "variantId": "v1",
    "variantName": "Control",
    "totalAssigned": 510,
    "totalSent": 500,
    "totalDelivered": 492,
    "totalConverted": 45,
    "conversionRate": 0.09,
    "isSignificant": false,
    "pValue": null,
    "confidenceInterval": null
  },
  {
    "variantId": "v2",
    "variantName": "Casual subject",
    "totalAssigned": 505,
    "totalSent": 500,
    "totalDelivered": 495,
    "totalConverted": 62,
    "conversionRate": 0.124,
    "isSignificant": true,
    "pValue": 0.032,
    "confidenceInterval": [0.002, 0.066]
  }
]
```

`conversionRate` is `totalConverted / totalSent`. Each non-control variant is compared with the control using a two-proportion z-test: `isSignificant` means p \< 0.05, and `confidenceInterval` is the 95% interval. The control keeps `pValue: null`. An unknown experiment id returns an empty `200`.

Conversions are [outcomes](https://molted.email/docs/analytics.md#ingest-outcomes) you record (`POST /v1/outcomes/ingest`, `molted outcomes ingest`, MCP `record_outcome`) that get attributed to a variant's send.

## Results in the portal

A person sees an experiment's results in the [portal](https://molted.email/docs/portal.md) on the page of the journey it runs in: `/app/journeys/<id>`, in an **Experiment** card under the journey's steps and runs. There is no separate Experiments page.

The card shows the experiment's name and, for each variant, how many contacts were **assigned**, how many emails were **sent**, and how many **converted**, with the conversion rate and a bar scaled to the best variant. Once anything has been sent, the variant with the highest conversion rate carries a **Leading** badge. That badge compares plain conversion rates; it is not a significance test. For `pValue`, `isSignificant` and the confidence interval, read [the results](#get-results) from the API.

The page shows one experiment per journey (the first it finds), and it only reads: creating, starting and stopping an experiment stay with your agents.

## How assignment works

Contacts are assigned to variants deterministically based on a SHA-256 hash of the experiment ID and contact email, and the first assignment is stored, so the same contact always sees the same variant, even across retries.

Holdout variants skip sending entirely. The decision is logged, but since nothing is sent, a holdout has no send for an outcome to be attributed to: its `totalSent` and `conversionRate` stay `0` in the results, so the results can't show holdout lift today.

## Journey integration

Experiments can be linked to a journey step via `journeyStepId` (see the field table above -- required in practice, not just in name). When a journey reaches a `send` step whose id matches a `running` experiment's `journeyStepId`, the contact is assigned to a variant. The variant's `subjectOverride` is applied to that send; `templateVersionId` is recorded but does not currently change what's rendered (see the caveat above, #1490).