# Send Diagnostics

Source: https://molted.email/docs/send-diagnostics

Read-only operator endpoints for diagnosing why a tenant's outgoing email is not sending, and what state an account is in.

When someone reports "our emails are not going out", the answer is usually one of five things: they were blocked by policy, deferred by domain warmup, held for approval, orphaned by a failed worker job, or attributed to a mailbox nobody was watching. The diagnostics endpoints answer each of those directly. When the report is about an account rather than a send ("I can't log in", "my dashboard is empty"), the [account lookups](#accounts) show whether the user exists, has a tenant, and can send.

All routes are `GET`, all queries are `SELECT`, and nothing here mutates tenant state. The only write is an audit row per call.

## Authentication

These routes accept `ADMIN_DIAGNOSTICS_KEY` as a bearer token:

```bash title="curl"
curl "https://api.molted.email/v1/admin/diagnostics/sends?tenantId=TENANT_ID" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"
```

Set `ADMIN_DIAGNOSTICS_KEY` to a value distinct from `ADMIN_API_KEY`. The diagnostics key unlocks only these read-only routes, so it can be handed to whoever (or whatever) is triaging an incident without also granting provisioning, billing, and quota mutation.

The routes **ship dark**: while `ADMIN_DIAGNOSTICS_KEY` is unset, every diagnostics route returns `404`. There is no fallback to `ADMIN_API_KEY`.

Each process allows 120 authenticated requests per minute and returns `429` beyond that. Every authorized call writes an `admin.diagnostics.read` row to `audit_events`, recording the route and only the `tenantId`, `window`, `limit`, `emailHash`, and `userId` parameters.

> **Warning:** Read-only does not mean low-sensitivity. These routes are **not tenant-scoped by credential**: `tenantId` is a parameter, so the key reads any tenant on the platform. No message bodies, subjects, payloads, names, session tokens, or key hashes are exposed, and email addresses are masked (see [Privacy](#privacy)), but treat the key as granting cross-tenant access to account metadata, not merely to counters.

The send routes require a `tenantId`. `/decisions` is the exception: `tenantId` is optional there (see [Decision shadow](#decision-shadow)), and [`/ownership`](#ownership) takes no parameters at all. `/sends`, `/blocks`, `/mailboxes` and `/decisions` take a `window` of `24h`, `7d`, or `30d`, defaulting to `7d`; `/warmup` and `/orphans` don't.

## Privacy

Response fields never hold a raw email address or a person's name:

- Emails are masked to the first character plus the domain: `m•••@acme.com`. The domain is kept because it matters for deliverability.
- Next to each masked email is an `emailHash`: the SHA-256 hex of `lower(trim(email))`. It is stable, so a hash from one response can be passed to the next call.
- Lookups take that hash, never the address, so no email lands in a URL, proxy log, or access log. Compute it on the client:

```bash title="shell"
printf '%s' 'someone@example.com' | tr '[:upper:]' '[:lower:]' | shasum -a 256 | cut -d' ' -f1
```

> **Warning:** Tenant IDs for accounts created before [#1317](https://github.com/chunkydotdev/molted-mail/issues/1317) contain the signup email's local part (`tenant-<local>-<id>`). Next to a masked email that can reveal the full address, so treat tenant IDs as personal data too.

> **Note:** **Breaking change:** `/orphans` items now return `recipientEmailMasked` and `recipientEmailHash` instead of `recipientEmail`, and `/mailboxes` returns `addressMasked` instead of `address`.

## Where to start

`GET /v1/admin/diagnostics/sends` returns the status histogram. The `status` and `deliveryStatus` pair is what separates a healthy send from a stuck one.

```json title="Response"
{
  "tenantId": "t_acme",
  "window": "7d",
  "total": 79,
  "byStatus": [
    { "status": "accepted", "deliveryStatus": "scheduled", "count": 63 },
    { "status": "blocked",  "deliveryStatus": null,        "count": 9 },
    { "status": "accepted", "deliveryStatus": "queued",    "count": 4 },
    { "status": "sent",     "deliveryStatus": "delivered", "count": 3 }
  ]
}
```

Read it like this:

| Pair | Meaning | Next step |
| - | - | - |
| `sent` / `delivered` | Healthy. | Nothing to do. |
| `accepted` / `queued` | In flight. | Check `/orphans` if the count is not draining. |
| `accepted` / `scheduled` | Deferred to a future date. | Check `/warmup`. |
| `pending_approval` | Held for human review. | Check the mailbox autonomy level via `/mailboxes`. |
| `blocked` | Rejected by policy. | Check `/blocks` for the reason. |

## Warmup

`GET /v1/admin/diagnostics/warmup` explains `scheduled` sends. A sender domain still in warmup has a daily cap, and non-transactional sends that exceed it are **deferred to a future date rather than failing** - the send reports success and goes out days later.

```json title="Response"
{
  "tenantId": "t_acme",
  "deferHorizonDays": 30,
  "domains": [
    {
      "domain": "mail.acme.com",
      "status": "verified",
      "tracked": true,
      "warmupOverride": false,
      "firstSendAt": "2026-08-01T19:34:17.017Z",
      "daysSinceFirstSend": 3,
      "dailyLimit": 100,
      "usedToday": 100,
      "remainingToday": 0,
      "futureReservations": [{ "slotDate": "2026-08-05", "reserved": 63, "dailyLimit": 100 }]
    }
  ],
  "deferredSends": {
    "total": 63,
    "scheduledTotal": 71,
    "reservedFutureSlots": 63,
    "earliestSlot": "2026-08-05",
    "latestSlot": "2026-08-05",
    "payloadEncrypted": false
  }
}
```

`remainingToday: 0` with a non-empty `futureReservations` is the signature of a tenant whose sends are silently piling up on future dates. A `dailyLimit` of `null` means the domain is exempt (either `warmupOverride` is set or the domain is not tracked). Each future slot carries its own `dailyLimit`, because a day further out may sit in a higher warmup tier than today.

`tracked: false` marks a domain seen only in the send counters and absent from `tenant_domains`, which is what a shared sending domain looks like. Warmup does not apply to those, so their `dailyLimit` is `null`, and their `status` (the domain's verification status otherwise) is `null` too.

### Reading `deferredSends` correctly

| Field | Meaning |
| - | - |
| `total` | Sends positively identified as warmup-deferred, or `null` when that cannot be determined |
| `scheduledTotal` | Every scheduled send, warmup-deferred or explicitly scheduled by the caller. Always accurate |
| `reservedFutureSlots` | Warmup slots reserved on future dates, summed across domains. Always accurate |
| `payloadEncrypted` | `true` when at least one scheduled row is encrypted, which forces `total`, `earliestSlot` and `latestSlot` to `null` |

The warmup markers live in `payload_json`, which is stored as `{}` on deployments with `CONTENT_ENCRYPTION_KEY` set. On those deployments `total` is `null` rather than `0`, since zero would read as a false all-clear. Use `scheduledTotal` and `reservedFutureSlots` there; both are derived from unencrypted columns.

Note the cap is per tenant and sender domain, not per mailbox. Tenants that provision one mailbox per end user share a single bucket across all of them.

## Block reasons

`GET /v1/admin/diagnostics/blocks` ranks policy rejections by frequency.

```json title="Response"
{
  "tenantId": "t_acme",
  "window": "7d",
  "total": 9,
  "reasons": [
    { "reason": "cooldown",   "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" },
    { "reason": "duplicate",  "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" },
    { "reason": "suppressed", "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" }
  ]
}
```

Most of these checks are **tenant-scoped rather than mailbox-scoped**: suppression, cooldown, dedupe, and rate limits all apply across the whole tenant. On a mailbox-per-end-user model, one user's activity can block another's. (#1398: a daily injection-risk budget used to be tenant-scoped too, and any outsider's inbound mail could exhaust it and block every other user's sends -- it's now a per-thread hold instead, scoped to the one thread being replied to.)

## Orphans

`GET /v1/admin/diagnostics/orphans` finds sends with no terminal state: rows claiming `delivery_status: "queued"` that never produced a send attempt, older than one hour. These are genuinely wedged - the worker job died before recording an attempt, so they will never send, never fail, and appear in no approval or override queue.

```bash title="curl"
curl "https://api.molted.email/v1/admin/diagnostics/orphans?tenantId=TENANT_ID&limit=100" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"
```

```json title="Response"
{
  "tenantId": "t_acme",
  "thresholdHours": 1,
  "total": 2,
  "oldestAgeHours": 5.2,
  "items": [
    {
      "requestId": "req_1",
      "recipientEmailMasked": "j•••@example.com",
      "recipientEmailHash": "9a1b...e4",
      "templateId": "welcome",
      "mailboxId": "mb_1",
      "status": "accepted",
      "deliveryStatus": "queued",
      "ageHours": 5.2,
      "createdAt": "2026-08-04T14:20:00.000Z"
    }
  ]
}
```

`total` is the full unpaginated count; `items` is capped by `limit` (default 50; a value above 500 is clamped to 500, and one below 1 or non-numeric is a `400 limit must be a positive integer`) and ordered oldest first. Each item identifies its recipient by `recipientEmailMasked` and `recipientEmailHash`.

## Per-mailbox breakdown

`GET /v1/admin/diagnostics/mailboxes` returns autonomy level, mailbox status, reputation, and send counts for each mailbox, plus `unattributedSends`, ordered by send count. `window` bounds both `sends` and `unattributedSends`.

```json title="Response"
{
  "tenantId": "t_acme",
  "mailboxes": [
    {
      "mailboxId": "mb_1",
      "addressMasked": "u•••@mail.acme.com",
      "status": "active",
      "autonomyLevel": 3,
      "reputationStatus": "healthy",
      "bounceRate": 0.01,
      "complaintRate": 0,
      "createdAt": "2026-07-20T09:00:00.000Z",
      "sends": { "total": 77, "queued": 4, "scheduled": 63, "pendingApproval": 0, "failed": 0, "blocked": 9 }
    }
  ],
  "unattributedSends": 2
}
```

A non-zero `unattributedSends` means sends arrived without a `mailboxId`. Those go out from the tenant's [default mailbox](https://molted.email/docs/mailboxes.md#default-mailbox) (the one you chose, or the oldest active mailbox when none is chosen or the chosen one is paused or deleted), so on a mailbox-per-user model they are being sent from the wrong identity and charged to the wrong reputation.

`autonomyLevel` of `1` or `2` explains any `pendingApproval` count - see [Autonomy Levels](https://molted.email/docs/autonomy-levels.md).

## Decision shadow

`GET /v1/admin/diagnostics/decisions` reports whether the Carapace/typed-decision shadow (#1401, epic #1400) is actually running, without SSH or raw SQL against `model_decisions`. `tenantId` is **optional** here: omit it to aggregate across every shadow-eligible tenant, which is the normal way to check "is the shadow running at all"; pass it to scope to one tenant.

```bash title="curl"
curl "https://api.molted.email/v1/admin/diagnostics/decisions?window=24h" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"
```

Rows are grouped by `decisionType`, `provider` (`keyword` | `jev` | `ensemble`), `modelVersion`, and `questionSetVersion`, plus `carapaceVersion` for `ensemble` rows (null for `keyword`/`jev` -- see `apps/worker/src/decision-shadow.ts`, which tags every ensemble row with the frozen Carapace tuning that produced it).

```json title="Response"
{
  "tenantId": null,
  "window": "24h",
  "groups": [
    {
      "decisionType": "inbound_classify",
      "provider": "ensemble",
      "modelVersion": "ensemble:jev-2026-08",
      "questionSetVersion": "inbound-safety-intent@4",
      "carapaceVersion": "v2.2",
      "rows": 118,
      "firstAt": "2026-09-27T00:01:00.000Z",
      "lastAt": "2026-09-28T00:01:00.000Z",
      "fallback": { "total": 5, "byReason": [{ "reason": "timeout", "count": 5 }] },
      "latencyMs": { "p50": 210, "p95": 890 },
      "verdictDistribution": [{ "verdict": "clean", "count": 100 }, { "verdict": "spam", "count": 18 }],
      "disagreesWithKeyword": 12,
      "raised": 14,
      "downgraded": 2,
      "brandGuardApplied": 9
    }
  ]
}
```

`verdictDistribution`, `disagreesWithKeyword`, `raised`, `downgraded`, and `brandGuardApplied` are only present on `provider: "ensemble"` groups. `disagreesWithKeyword` pairs each ensemble row with its sibling `keyword` row for the same subject and counts where the verdicts differ -- the shadow-vs-live gap the epic #1400 promotion rule cares about.

> **Warning:** This route returns aggregates only -- counts, timestamps, and closed-vocabulary labels (verdicts, fallback reasons). It never returns the `answers`/`probabilities` JSONB, `subject_id`, or anything else that traces back to a specific inbound message.

### Fallback reasons are bucketed, never raw

The underlying `model_decisions.fallback_reason` column is free text -- it is an upstream error message (`err.message`, or a wrapper around it) written by `jev-provider.ts` and `outbound-content.ts`, and could otherwise carry anything from an HTTP status to a fragment of an upstream response. `fallback.byReason[].reason` is never that raw text: the query buckets it in SQL into one of a fixed set of categories before it is ever selected, so every distinct upstream message collapses into one of:

| Bucket | Covers |
| - | - |
| `keyword_fallback_failed` | Jev failed AND the keyword fallback it fell back to also threw -- the worst case |
| `not_configured` | No `TYPESAFE_API_KEY`/pinned `JEV_MODEL` configured |
| `timeout` | The request timed out, hit its deadline, or was aborted |
| `auth` | `401`/`403` from TypeSafe, or an authorization/forbidden error |
| `rate_limited` | `429`, or a rate-limit/quota message |
| `upstream_5xx` | Any other `5xx` from TypeSafe |
| `upstream_4xx` | Any other `4xx` from TypeSafe |
| `network` | A network-level failure (connection refused, DNS, socket, fetch failed) |
| `parse` | An invalid/malformed response or answer TypeSafe returned |
| `other` | Anything that matches none of the above |

An empty `groups` array most often means the tenant isn't yet eligible under the rollout-date rule (#2322: new tenants from 2026-10-10, every tenant from 2026-11-09, early access via `DECISION_SHADOW_TENANTS` until then), or `TYPESAFE_API_KEY`/`JEV_MODEL` are unset -- see `isShadowEligible` in `decision-shadow.ts` and `isSecurityModelEligible`/`resolveSecurityModelEligible` in `packages/mail-policy/src/decision-model/security-model-eligibility.ts`.

## Accounts

### Users

`GET /v1/admin/diagnostics/users` looks up one account. Pass exactly one of `emailHash` (see [Privacy](#privacy)) or `userId`; anything else is a `400`, and no match is a `404`.

```bash title="curl"
curl "https://api.molted.email/v1/admin/diagnostics/users?emailHash=EMAIL_HASH" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"
```

```json title="Response"
{
  "user": {
    "id": "u_123",
    "emailMasked": "m•••@acme.com",
    "emailHash": "5f0e...c2",
    "emailVerified": true,
    "createdAt": "2026-09-01T10:00:00.000Z",
    "suspendedAt": null,
    "hasPassword": true,
    "lastActiveAt": "2026-09-22T08:14:00.000Z",
    "activeSessions": 1,
    "magicLinksLast30d": 2
  },
  "memberships": [],
  "flags": { "noTenant": true, "tenantDeletedTombstone": false, "suspended": false }
}
```

| Flag | Meaning |
| - | - |
| `noTenant` | The user belongs to no tenant. Signing in normally provisions one, so this usually means provisioning failed. |
| `tenantDeletedTombstone` | An admin deleted the user's tenant at some point. Since #2046 this no longer blocks sign-in: tenant deletion erases a user with no other tenant, and a user who is still here gets a fresh tenant on sign-in, provisioned without trial eligibility. |
| `suspended` | The account is suspended (`suspendedAt` is set). A suspended account cannot sign in and its existing sessions are rejected with `403 USER_SUSPENDED` on their next request. |

`hasPassword` is false for accounts that only ever signed in by magic link. `lastActiveAt` is the most recent session activity. Each `memberships` item is `{ tenantId, role, tenantStatus, billingPlan, joinedAt }`; `tenantStatus` is `null` if the tenant row is gone.

### Tenants

`GET /v1/admin/diagnostics/tenants/:id` returns the tenant's billing and setup state, with `404` for an unknown id.

```json title="Response"
{
  "tenant": {
    "id": "t_acme",
    "status": "active",
    "billingPlan": "trial",
    "trialEndsAt": "2026-10-05T10:00:00.000Z",
    "graceUntil": null,
    "hasPaymentMethod": false,
    "termsAcceptedAt": "2026-09-21T10:00:00.000Z",
    "deletionScheduledFor": null,
    "createdAt": "2026-09-21T10:00:00.000Z"
  },
  "sendBlocked": true,
  "sendBlockReason": "trial_not_activated",
  "members": [{ "userId": "u_123", "role": "owner", "emailMasked": "m•••@acme.com", "emailHash": "5f0e...c2" }],
  "mailboxes": [{ "id": "mb_1", "addressMasked": "a•••@mail.acme.com", "status": "active", "autonomyLevel": 3 }],
  "domains": [{ "domain": "mail.acme.com", "status": "verified", "receivingEnabled": true, "firstSendAt": null }],
  "apiKeys": [{ "id": "k_1", "keyPrefix": "mm_live_ab", "label": "default", "status": "active", "agentId": null, "lastUsedAt": null, "revokedAt": null, "createdAt": "2026-09-21T10:00:00.000Z" }],
  "quotas": { "dailySendLimit": 50, "hourlySendLimit": 20, "monthlySendLimit": 100, "maxMailboxes": 3, "maxDomains": 0 }
}
```

`sendBlocked` and `sendBlockReason` (`trial_not_activated`, `subscription_expired` or `null`) use the same billing check as the send path, so they say exactly whether the next send would be rejected for billing. `quotas` is `null` when the tenant has no quota row. The tenant name is omitted because it defaults to the owner's name, and API keys are listed by prefix only.

### Ownership

`GET /v1/admin/diagnostics/ownership` takes no parameters and reports workspaces no person owns and recent agent login tokens (#1337):

```json title="Response"
{
  "workspacesWithoutHumanOwner": { "total": 12, "excludingTest": 9 },
  "loginTokensLast30d": {
    "total": 30,
    "byUser": [
      { "userId": "u_123", "tokens": 4, "lastMintedAt": "2026-09-22T08:14:00.000Z", "ownedWorkspaces": [{ "tenantId": "t_acme", "hasHumanOwner": false }] }
    ]
  },
  "claimLinksLast30d": { "issued": 20, "redeemed": 7 }
}
```

`byUser` is capped at 200 entries.

### Cross-tenant experiments

`GET /v1/admin/diagnostics/experiments/cross-tenant` takes no parameters and lists experiments whose journey, journey step or segment belongs to a different workspace than the experiment (#2424; created before the API checked ownership). Read-only; ids, tenant ids and status only:

```json title="Response"
{
  "count": 1,
  "rows": [
    { "experimentId": "e_1", "experimentTenantId": "t_a", "journeyTenantId": "t_b", "stepJourneyTenantId": null, "segmentTenantId": null, "status": "running" }
  ]
}
```

Each tenant field is set only where it differs from the experiment's own tenant. `count` is the total; `rows` is capped at 100, newest first. Older tenant ids embed an email local part (#1317), so treat the response as personal data.

### Unbacked inbound messages

`GET /v1/admin/diagnostics/inbound/unbacked` takes no parameters and lists inbound conversation messages that no received email stands behind (#2533). Before #2533, any agent key could write a message into a conversation as if it had arrived, through the agentic-mailbox's inbound projection, so these are the messages to review. Each row has one of four reasons:

- `orphaned`: its inbound message id names no received message.
- `cross_tenant`: the id names another workspace's message.
- `sender_mismatch`: the id names a message of the same workspace, but that message came from a different sender.
- `provider_mismatch`: the id names a message of the same workspace from the same sender, but the received message has a provider message id and the conversation message doesn't carry it.

`orphaned` has one known false positive. Deleting a mailbox for good (the purge after its restore window) removes its received messages. A message of it that was filed into another mailbox's conversation stays behind, pointing at nothing. The purge keeps no list of what it removed, so the check can't rule this out. Instead, a row is marked `purgeExplainable` when its workspace purged a mailbox after the message arrived, and `orphanedPurgeExplainable` counts those rows.

Read-only; ids, tenant, mailbox, time and state only, never an address or content. The query runs under a 15-second statement timeout:

```json title="Response"
{
  "count": 1,
  "byReason": { "orphaned": 1, "cross_tenant": 0, "sender_mismatch": 0, "provider_mismatch": 0 },
  "orphanedPurgeExplainable": 0,
  "withoutInboundMessageId": 0,
  "rows": [
    { "threadMessageId": "tm_1", "threadId": "th_1", "tenantId": "t_a", "mailboxId": "mb_1", "inboundMessageId": "im_1", "reason": "orphaned", "purgeExplainable": false, "createdAt": "2026-10-20T09:00:00.000Z", "quarantined": false, "held": false }
  ]
}
```

`count` is the total over the four reasons; `rows` is capped at 100, newest first. `withoutInboundMessageId` counts inbound messages with no inbound message id at all. Those are either old rows from before the id was required or messages planted before then, so they are counted apart rather than listed. Older tenant ids embed an email local part (#1317), so treat the response as personal data.

## MCP tools

With `MOLTED_ADMIN_KEY` set, the [MCP server](https://molted.email/docs/mcp-server.md) exposes these as `diagnose_sends`, `diagnose_warmup`, `diagnose_blocks`, `diagnose_orphans` (takes `limit`), `diagnose_mailboxes`, `diagnose_user`, and `diagnose_tenant`. There are no MCP tools for `/decisions` or `/ownership`. `diagnose_user` takes exactly one of `email`, `emailHash` or `userId`; a plain `email` is hashed inside the tool, so the address never leaves the MCP process. `MOLTED_ADMIN_API_URL` defaults to `https://api.molted.email`. Without that variable the tools are not registered, so normal agent runtimes never see them.

> **Warning:** These tools are registered on the **`stdio` transport only**, run from the repo, and never on the hosted endpoint at `mcp.molted.email`. The HTTP transport is a shared endpoint where each caller authenticates with their own tenant API key, while these tools run with the operator key and take `tenantId` as a caller-supplied argument, so registering them there would let any tenant read any other tenant's data. Running `--transport http` with `MOLTED_ADMIN_KEY` set logs a warning and omits the tools.

For a tenant's own view of the same questions, with a normal API key, use the MCP `check_send_health` tool (`mailbox_id` optional) or `molted doctor [--mailbox <id>] [--json]`.

Build the server in a checkout of the repo (`pnpm --filter @molted/mcp build`), then point a local stdio config at the built entry point:

```json title="claude_desktop_config.json"
{
  "mcpServers": {
    "molted-ops": {
      "command": "node",
      "args": ["/path/to/molted-mail/packages/mcp/dist/index.js"],
      "env": {
        "MOLTED_API_KEY": "mm_live_...",
        "MOLTED_ADMIN_KEY": "your_diagnostics_key",
        "MOLTED_ADMIN_API_URL": "https://api.molted.email"
      }
    }
  }
}
```