Send Diagnostics

When someone reports "our emails are not going out", the answer is usually one of five things: they were blocked by policy, deferred by domain warmup, held for approval, orphaned by a failed worker job, or attributed to a mailbox nobody was watching. The diagnostics endpoints answer each of those directly. When the report is about an account rather than a send ("I can't log in", "my dashboard is empty"), the account lookups show whether the user exists, has a tenant, and can send.

All routes are GET, all queries are SELECT, and nothing here mutates tenant state. The only write is an audit row per call.

Authentication

These routes accept ADMIN_DIAGNOSTICS_KEY as a bearer token:

curl "https://api.molted.email/v1/admin/diagnostics/sends?tenantId=TENANT_ID" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"

Set ADMIN_DIAGNOSTICS_KEY to a value distinct from ADMIN_API_KEY. The diagnostics key unlocks only these read-only routes, so it can be handed to whoever (or whatever) is triaging an incident without also granting provisioning, billing, and quota mutation.

The routes ship dark: while ADMIN_DIAGNOSTICS_KEY is unset, every diagnostics route returns 404. There is no fallback to ADMIN_API_KEY.

Each process allows 120 authenticated requests per minute and returns 429 beyond that. Every authorized call writes an admin.diagnostics.read row to audit_events, recording the route and only the tenantId, window, limit, emailHash, and userId parameters.

The send routes require a tenantId. /decisions is the exception: tenantId is optional there (see Decision shadow), and /ownership takes no parameters at all. /sends, /blocks, /mailboxes and /decisions take a window of 24h, 7d, or 30d, defaulting to 7d; /warmup and /orphans don't.

Privacy

Response fields never hold a raw email address or a person's name:

• Emails are masked to the first character plus the domain: m•••@acme.com. The domain is kept because it matters for deliverability.
• Next to each masked email is an emailHash: the SHA-256 hex of lower(trim(email)). It is stable, so a hash from one response can be passed to the next call.
• Lookups take that hash, never the address, so no email lands in a URL, proxy log, or access log. Compute it on the client:

printf '%s' 'someone@example.com' | tr '[:upper:]' '[:lower:]' | shasum -a 256 | cut -d' ' -f1

Where to start

GET /v1/admin/diagnostics/sends returns the status histogram. The status and deliveryStatus pair is what separates a healthy send from a stuck one.

{
  "tenantId": "t_acme",
  "window": "7d",
  "total": 79,
  "byStatus": [
    { "status": "accepted", "deliveryStatus": "scheduled", "count": 63 },
    { "status": "blocked",  "deliveryStatus": null,        "count": 9 },
    { "status": "accepted", "deliveryStatus": "queued",    "count": 4 },
    { "status": "sent",     "deliveryStatus": "delivered", "count": 3 }
  ]
}

Read it like this:

Warmup

GET /v1/admin/diagnostics/warmup explains scheduled sends. A sender domain still in warmup has a daily cap, and non-transactional sends that exceed it are deferred to a future date rather than failing - the send reports success and goes out days later.

{
  "tenantId": "t_acme",
  "deferHorizonDays": 30,
  "domains": [
    {
      "domain": "mail.acme.com",
      "status": "verified",
      "tracked": true,
      "warmupOverride": false,
      "firstSendAt": "2026-08-01T19:34:17.017Z",
      "daysSinceFirstSend": 3,
      "dailyLimit": 100,
      "usedToday": 100,
      "remainingToday": 0,
      "futureReservations": [{ "slotDate": "2026-08-05", "reserved": 63, "dailyLimit": 100 }]
    }
  ],
  "deferredSends": {
    "total": 63,
    "scheduledTotal": 71,
    "reservedFutureSlots": 63,
    "earliestSlot": "2026-08-05",
    "latestSlot": "2026-08-05",
    "payloadEncrypted": false
  }
}

remainingToday: 0 with a non-empty futureReservations is the signature of a tenant whose sends are silently piling up on future dates. A dailyLimit of null means the domain is exempt (either warmupOverride is set or the domain is not tracked). Each future slot carries its own dailyLimit, because a day further out may sit in a higher warmup tier than today.

tracked: false marks a domain seen only in the send counters and absent from tenant_domains, which is what a shared sending domain looks like. Warmup does not apply to those, so their dailyLimit is null, and their status (the domain's verification status otherwise) is null too.

Reading deferredSends correctly

The warmup markers live in payload_json, which is stored as {} on deployments with CONTENT_ENCRYPTION_KEY set. On those deployments total is null rather than 0, since zero would read as a false all-clear. Use scheduledTotal and reservedFutureSlots there; both are derived from unencrypted columns.

Note the cap is per tenant and sender domain, not per mailbox. Tenants that provision one mailbox per end user share a single bucket across all of them.

Block reasons

GET /v1/admin/diagnostics/blocks ranks policy rejections by frequency.

{
  "tenantId": "t_acme",
  "window": "7d",
  "total": 9,
  "reasons": [
    { "reason": "cooldown",   "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" },
    { "reason": "duplicate",  "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" },
    { "reason": "suppressed", "count": 3, "lastSeenAt": "2026-08-04T19:34:17.017Z" }
  ]
}

Most of these checks are tenant-scoped rather than mailbox-scoped: suppression, cooldown, dedupe, and rate limits all apply across the whole tenant. On a mailbox-per-end-user model, one user's activity can block another's. (#1398: a daily injection-risk budget used to be tenant-scoped too, and any outsider's inbound mail could exhaust it and block every other user's sends -- it's now a per-thread hold instead, scoped to the one thread being replied to.)

Orphans

GET /v1/admin/diagnostics/orphans finds sends with no terminal state: rows claiming delivery_status: "queued" that never produced a send attempt, older than one hour. These are genuinely wedged - the worker job died before recording an attempt, so they will never send, never fail, and appear in no approval or override queue.

curl "https://api.molted.email/v1/admin/diagnostics/orphans?tenantId=TENANT_ID&limit=100" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"

{
  "tenantId": "t_acme",
  "thresholdHours": 1,
  "total": 2,
  "oldestAgeHours": 5.2,
  "items": [
    {
      "requestId": "req_1",
      "recipientEmailMasked": "j•••@example.com",
      "recipientEmailHash": "9a1b...e4",
      "templateId": "welcome",
      "mailboxId": "mb_1",
      "status": "accepted",
      "deliveryStatus": "queued",
      "ageHours": 5.2,
      "createdAt": "2026-08-04T14:20:00.000Z"
    }
  ]
}

total is the full unpaginated count; items is capped by limit (default 50; a value above 500 is clamped to 500, and one below 1 or non-numeric is a 400 limit must be a positive integer) and ordered oldest first. Each item identifies its recipient by recipientEmailMasked and recipientEmailHash.

Per-mailbox breakdown

GET /v1/admin/diagnostics/mailboxes returns autonomy level, mailbox status, reputation, and send counts for each mailbox, plus unattributedSends, ordered by send count. window bounds both sends and unattributedSends.

{
  "tenantId": "t_acme",
  "mailboxes": [
    {
      "mailboxId": "mb_1",
      "addressMasked": "u•••@mail.acme.com",
      "status": "active",
      "autonomyLevel": 3,
      "reputationStatus": "healthy",
      "bounceRate": 0.01,
      "complaintRate": 0,
      "createdAt": "2026-07-20T09:00:00.000Z",
      "sends": { "total": 77, "queued": 4, "scheduled": 63, "pendingApproval": 0, "failed": 0, "blocked": 9 }
    }
  ],
  "unattributedSends": 2
}

A non-zero unattributedSends means sends arrived without a mailboxId. Those go out from the tenant's default mailbox (the one you chose, or the oldest active mailbox when none is chosen or the chosen one is paused or deleted), so on a mailbox-per-user model they are being sent from the wrong identity and charged to the wrong reputation.

autonomyLevel of 1 or 2 explains any pendingApproval count - see Autonomy Levels.

Decision shadow

GET /v1/admin/diagnostics/decisions reports whether the Carapace/typed-decision shadow (#1401, epic #1400) is actually running, without SSH or raw SQL against model_decisions. tenantId is optional here: omit it to aggregate across every shadow-eligible tenant, which is the normal way to check "is the shadow running at all"; pass it to scope to one tenant.

curl "https://api.molted.email/v1/admin/diagnostics/decisions?window=24h" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"

Rows are grouped by decisionType, provider (keyword | jev | ensemble), modelVersion, and questionSetVersion, plus carapaceVersion for ensemble rows (null for keyword/jev -- see apps/worker/src/decision-shadow.ts, which tags every ensemble row with the frozen Carapace tuning that produced it).

{
  "tenantId": null,
  "window": "24h",
  "groups": [
    {
      "decisionType": "inbound_classify",
      "provider": "ensemble",
      "modelVersion": "ensemble:jev-2026-08",
      "questionSetVersion": "inbound-safety-intent@4",
      "carapaceVersion": "v2.2",
      "rows": 118,
      "firstAt": "2026-09-27T00:01:00.000Z",
      "lastAt": "2026-09-28T00:01:00.000Z",
      "fallback": { "total": 5, "byReason": [{ "reason": "timeout", "count": 5 }] },
      "latencyMs": { "p50": 210, "p95": 890 },
      "verdictDistribution": [{ "verdict": "clean", "count": 100 }, { "verdict": "spam", "count": 18 }],
      "disagreesWithKeyword": 12,
      "raised": 14,
      "downgraded": 2,
      "brandGuardApplied": 9
    }
  ]
}

verdictDistribution, disagreesWithKeyword, raised, downgraded, and brandGuardApplied are only present on provider: "ensemble" groups. disagreesWithKeyword pairs each ensemble row with its sibling keyword row for the same subject and counts where the verdicts differ -- the shadow-vs-live gap the epic #1400 promotion rule cares about.

Fallback reasons are bucketed, never raw

The underlying model_decisions.fallback_reason column is free text -- it is an upstream error message (err.message, or a wrapper around it) written by jev-provider.ts and outbound-content.ts, and could otherwise carry anything from an HTTP status to a fragment of an upstream response. fallback.byReason[].reason is never that raw text: the query buckets it in SQL into one of a fixed set of categories before it is ever selected, so every distinct upstream message collapses into one of:

An empty groups array most often means the tenant isn't yet eligible under the rollout-date rule (#2322: new tenants from 2026-10-10, every tenant from 2026-11-09, early access via DECISION_SHADOW_TENANTS until then), or TYPESAFE_API_KEY/JEV_MODEL are unset -- see isShadowEligible in decision-shadow.ts and isSecurityModelEligible/resolveSecurityModelEligible in packages/mail-policy/src/decision-model/security-model-eligibility.ts.

Accounts

Users

GET /v1/admin/diagnostics/users looks up one account. Pass exactly one of emailHash (see Privacy) or userId; anything else is a 400, and no match is a 404.

curl "https://api.molted.email/v1/admin/diagnostics/users?emailHash=EMAIL_HASH" \
  -H "Authorization: Bearer YOUR_DIAGNOSTICS_KEY"

{
  "user": {
    "id": "u_123",
    "emailMasked": "m•••@acme.com",
    "emailHash": "5f0e...c2",
    "emailVerified": true,
    "createdAt": "2026-09-01T10:00:00.000Z",
    "suspendedAt": null,
    "hasPassword": true,
    "lastActiveAt": "2026-09-22T08:14:00.000Z",
    "activeSessions": 1,
    "magicLinksLast30d": 2
  },
  "memberships": [],
  "flags": { "noTenant": true, "tenantDeletedTombstone": false, "suspended": false }
}

hasPassword is false for accounts that only ever signed in by magic link. lastActiveAt is the most recent session activity. Each memberships item is { tenantId, role, tenantStatus, billingPlan, joinedAt }; tenantStatus is null if the tenant row is gone.

Tenants

GET /v1/admin/diagnostics/tenants/:id returns the tenant's billing and setup state, with 404 for an unknown id.

{
  "tenant": {
    "id": "t_acme",
    "status": "active",
    "billingPlan": "trial",
    "trialEndsAt": "2026-10-05T10:00:00.000Z",
    "graceUntil": null,
    "hasPaymentMethod": false,
    "termsAcceptedAt": "2026-09-21T10:00:00.000Z",
    "deletionScheduledFor": null,
    "createdAt": "2026-09-21T10:00:00.000Z"
  },
  "sendBlocked": true,
  "sendBlockReason": "trial_not_activated",
  "members": [{ "userId": "u_123", "role": "owner", "emailMasked": "m•••@acme.com", "emailHash": "5f0e...c2" }],
  "mailboxes": [{ "id": "mb_1", "addressMasked": "a•••@mail.acme.com", "status": "active", "autonomyLevel": 3 }],
  "domains": [{ "domain": "mail.acme.com", "status": "verified", "receivingEnabled": true, "firstSendAt": null }],
  "apiKeys": [{ "id": "k_1", "keyPrefix": "mm_live_ab", "label": "default", "status": "active", "agentId": null, "lastUsedAt": null, "revokedAt": null, "createdAt": "2026-09-21T10:00:00.000Z" }],
  "quotas": { "dailySendLimit": 50, "hourlySendLimit": 20, "monthlySendLimit": 100, "maxMailboxes": 3, "maxDomains": 0 }
}

sendBlocked and sendBlockReason (trial_not_activated, subscription_expired or null) use the same billing check as the send path, so they say exactly whether the next send would be rejected for billing. quotas is null when the tenant has no quota row. The tenant name is omitted because it defaults to the owner's name, and API keys are listed by prefix only.

Ownership

GET /v1/admin/diagnostics/ownership takes no parameters and reports workspaces no person owns and recent agent login tokens (#1337):

{
  "workspacesWithoutHumanOwner": { "total": 12, "excludingTest": 9 },
  "loginTokensLast30d": {
    "total": 30,
    "byUser": [
      { "userId": "u_123", "tokens": 4, "lastMintedAt": "2026-09-22T08:14:00.000Z", "ownedWorkspaces": [{ "tenantId": "t_acme", "hasHumanOwner": false }] }
    ]
  },
  "claimLinksLast30d": { "issued": 20, "redeemed": 7 }
}

byUser is capped at 200 entries.

Cross-tenant experiments

GET /v1/admin/diagnostics/experiments/cross-tenant takes no parameters and lists experiments whose journey, journey step or segment belongs to a different workspace than the experiment (#2424; created before the API checked ownership). Read-only; ids, tenant ids and status only:

{
  "count": 1,
  "rows": [
    { "experimentId": "e_1", "experimentTenantId": "t_a", "journeyTenantId": "t_b", "stepJourneyTenantId": null, "segmentTenantId": null, "status": "running" }
  ]
}

Each tenant field is set only where it differs from the experiment's own tenant. count is the total; rows is capped at 100, newest first. Older tenant ids embed an email local part (#1317), so treat the response as personal data.

Unbacked inbound messages

GET /v1/admin/diagnostics/inbound/unbacked takes no parameters and lists inbound conversation messages that no received email stands behind (#2533). Before #2533, any agent key could write a message into a conversation as if it had arrived, through the agentic-mailbox's inbound projection, so these are the messages to review. Each row has one of four reasons:

• orphaned: its inbound message id names no received message.
• cross_tenant: the id names another workspace's message.
• sender_mismatch: the id names a message of the same workspace, but that message came from a different sender.
• provider_mismatch: the id names a message of the same workspace from the same sender, but the received message has a provider message id and the conversation message doesn't carry it.

orphaned has one known false positive. Deleting a mailbox for good (the purge after its restore window) removes its received messages. A message of it that was filed into another mailbox's conversation stays behind, pointing at nothing. The purge keeps no list of what it removed, so the check can't rule this out. Instead, a row is marked purgeExplainable when its workspace purged a mailbox after the message arrived, and orphanedPurgeExplainable counts those rows.

Read-only; ids, tenant, mailbox, time and state only, never an address or content. The query runs under a 15-second statement timeout:

{
  "count": 1,
  "byReason": { "orphaned": 1, "cross_tenant": 0, "sender_mismatch": 0, "provider_mismatch": 0 },
  "orphanedPurgeExplainable": 0,
  "withoutInboundMessageId": 0,
  "rows": [
    { "threadMessageId": "tm_1", "threadId": "th_1", "tenantId": "t_a", "mailboxId": "mb_1", "inboundMessageId": "im_1", "reason": "orphaned", "purgeExplainable": false, "createdAt": "2026-10-20T09:00:00.000Z", "quarantined": false, "held": false }
  ]
}

count is the total over the four reasons; rows is capped at 100, newest first. withoutInboundMessageId counts inbound messages with no inbound message id at all. Those are either old rows from before the id was required or messages planted before then, so they are counted apart rather than listed. Older tenant ids embed an email local part (#1317), so treat the response as personal data.

MCP tools

With MOLTED_ADMIN_KEY set, the MCP server exposes these as diagnose_sends, diagnose_warmup, diagnose_blocks, diagnose_orphans (takes limit), diagnose_mailboxes, diagnose_user, and diagnose_tenant. There are no MCP tools for /decisions or /ownership. diagnose_user takes exactly one of email, emailHash or userId; a plain email is hashed inside the tool, so the address never leaves the MCP process. MOLTED_ADMIN_API_URL defaults to https://api.molted.email. Without that variable the tools are not registered, so normal agent runtimes never see them.

For a tenant's own view of the same questions, with a normal API key, use the MCP check_send_health tool (mailbox_id optional) or molted doctor [--mailbox <id>] [--json].

Build the server in a checkout of the repo (pnpm --filter @molted/mcp build), then point a local stdio config at the built entry point:

{
  "mcpServers": {
    "molted-ops": {
      "command": "node",
      "args": ["/path/to/molted-mail/packages/mcp/dist/index.js"],
      "env": {
        "MOLTED_API_KEY": "mm_live_...",
        "MOLTED_ADMIN_KEY": "your_diagnostics_key",
        "MOLTED_ADMIN_API_URL": "https://api.molted.email"
      }
    }
  }
}