--- name: unified-debug-webhooks-connections description: >- Diagnose and fix unhealthy Unified.to connections and webhooks: interpret connection health statuses, read API call logs, resolve 401/403/404/429/5xx errors, validate webhook signatures, and understand native vs virtual webhook retry behavior. Use when a connection stops returning data, a webhook is not firing or is marked unhealthy, or webhook signature validation is failing. license: MIT metadata: category: reference api_base: https://api.unified.to docs: https://docs.unified.to/reference/webhooks --- # Debug Unified.to webhooks & connections When an integration stops working, the cause is almost always an unhealthy **connection** (credentials/scopes) or an unhealthy **webhook** (delivery or read failure). This skill teaches an agent how to diagnose and fix both, based on Unified.to's troubleshooting guides. ## When to use this skill - A connection is marked **Unhealthy** / **Unhealthy now**, or stopped returning data - A webhook is not firing, is delayed, or is marked **unhealthy** - Webhook **signature validation** is failing - You need to decide between polling, native webhooks, and virtual webhooks ## First: is it the connection or the webhook? A webhook can only be as healthy as the connection it reads from. **Check the connection first** — 401/403 errors reported against a webhook are really connection problems. Fix the connection, and the webhook usually recovers. --- ## Part 1 — Debugging unhealthy connections A connection is **unhealthy** when a provider no longer accepts its credentials or access. Transient issues (rate limits, temporary 5xx) do **not** flip health. ### Health statuses | Status | Meaning | | --------------- | ------------------------------------------------------------------- | | `Healthy` | A recent API call or OAuth token refresh succeeded | | `Unhealthy` | Provider rejected credentials/access; no success since | | `Unhealthy now` | Recently failed after previously succeeding | | `New` | Created but not yet used | ### Diagnose 1. Open the **Connections** dashboard in your Unified.to workspace. 2. Find the connection marked `Unhealthy` / `Unhealthy now` and note its **connection ID**. 3. Open **API Call Logs** and filter by that connection ID (logs are retained for the **past 60 days**). 4. Open the latest failing call and read its **status code** and description. ### Fix by status code | Code | Cause | Fix | | ------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | **401** | App access revoked, or the customer entered wrong credentials | Verify access, confirm credentials, then **recreate the connection** (re-authorize). | | **403** | Scope mismatch between the provider app and Unified.to | Make the provider developer-app scopes match Unified.to; update your embedded/auth URL scopes. | | **404** | Missing/incorrect input data (note: Enrichment treats 404 as success) | Validate the input data you sent. | | **429** | Provider rate limit exceeded | Throttle, switch to webhooks for sync, or request higher limits. | | **5xx** | Temporary provider outage or bug | Check the provider status page, retry with backoff, contact support if it persists. | ### Detect proactively Subscribe to the **Notifications** webhook (Workspace Settings) for real-time `CONNECTION_UNHEALTHY` and `CONNECTION_HEALTHY` events so you can prompt the end-user to re-authorize before they notice data is stale. See https://docs.unified.to/guides/how_to_troubleshoot_unhealthy_connections. --- ## Part 2 — Debugging unhealthy webhooks ### Failure modes | Failure | Meaning | | -------------------------- | ------------------------------------------------------------------------- | | **Failed to subscribe** | Creation failed — integration unavailable, missing permissions, or a bug (native webhooks only) | | **Failed to refresh token**| Could not reach the provider to retrieve event data | | **Failed to read** | Could not retrieve new data from the provider (config/integration error) | | **Failed to dispatch data**| **Your** server did not respond, or returned a non-2xx, to the webhook POST | | **Failed to process data** | A problem in the read or dispatch step | ### Diagnose 1. Open the **Webhooks** section in your workspace and note the webhook's **connection ID**. 2. Open **API Call Logs**, filter by that connection ID, and open the latest error to read the status code and description (60-day retention). Common codes: **400** = integration-level issue (needs vendor support); **401/403** = connection credential/scope problems (see Part 1), not the webhook itself. ### Retry behavior (know this before you "lose" an event) - **Your endpoint unavailable:** up to **3 immediate retries** (1s apart), then **Fibonacci backoff** (1, 2, 3, 5, 8 … minutes) continuing for **up to 2 weeks**. - **Provider unavailable:** Fibonacci backoff starting at 1-minute delays. So a temporarily failing endpoint is not a lost event — make your handler **idempotent** and return `200 OK` quickly. Do heavy work asynchronously. ### Detect proactively Subscribe to the **Notifications** webhook for the `WEBHOOK_UNHEALTHY` event. --- ## Part 3 — Webhook signature validation (a top cause of "webhooks not working") If your endpoint rejects valid deliveries, signature mismatch is usually why. The `sig256` field is `HMAC-SHA256(workspace_secret, data + nonce)`, base64 encoded — where `data` is the exact `data` array from the payload body and `nonce` is its string. (`sig` is the deprecated SHA-1 variant; use `sig256`.) Rules that break signatures if ignored: 1. **Preserve field order** of `data` exactly as received. In Go, unmarshal `data` as `json.RawMessage` — a `map[string]interface{}` reorders keys. 2. **No extra whitespace.** Re-serialize compactly. In Python use `json.dumps(data['data'], separators=(',', ':'), ensure_ascii=False)`. 3. Use a **timing-safe** comparison (`compare_digest` / `timingSafeEqual` / `subtle.ConstantTimeCompare`). 4. The **workspace secret** (app.unified.to → Settings → API) is server-side only — never validate in the browser. ```typescript import { createHmac, timingSafeEqual } from 'crypto'; function validateWebhook(workspaceSecret: string, body: { data: any[]; nonce: string; sig256: string }) { const computed = Buffer.from( createHmac('sha256', workspaceSecret) .update(JSON.stringify(body.data)) // must match the sender's encoding/order .update(body.nonce) .digest('base64'), 'utf8' ); const provided = Buffer.from(body.sig256, 'utf8'); return computed.length === provided.length && timingSafeEqual(computed, provided); } ``` ### Data webhook payload fields `data` (array), `webhook`, `nonce`, `sig` (deprecated SHA-1), `sig256`, `external_xref` (the value you set on the connection, echoed back), and `type`: - `INITIAL-PARTIAL` — each page of the initial sync - `INITIAL-COMPLETE` — last page (or empty list) of the initial sync - `VIRTUAL` — each page from a virtual webhook - `NATIVE` — each page from a native webhook --- ## Part 4 — Native vs virtual webhooks (when "it never fires") If an integration has no native webhook support, use a **virtual webhook** — Unified.to polls the provider on an interval you set and posts changes to your URL, so you consume it exactly like a native webhook. | | Native | Virtual | | --------------- | ----------------------------------- | ---------------------------------------------- | | Delivery | Immediate push | Scheduled polling (≥ ~1 min), interval you set | | Events | created, updated, deleted (if supported) | mainly created & updated | | Retry / health | per-integration | built-in read + dispatch retry & health tracking | Check support: integration page → **Feature Support** → **Webhooks** (rows are labeled `native` or `virtual`). If you expected instant delivery but the integration is virtual-only, the delay is expected — lower the interval or accept the latency. See https://docs.unified.to/concepts/virtual_webhooks. --- ## Debug checklist 1. Reproduce and capture the exact **status code** from API Call Logs (filter by connection ID). 2. `401/403` → fix the **connection** (credentials/scopes), recreate it, retry. 3. Webhook not firing → confirm the integration supports the event type (native vs virtual) and that the connection is healthy. 4. Deliveries rejected → verify **signature** (order, whitespace, secret) and that your endpoint returns `200` fast and is idempotent. 5. `429`/`5xx` → transient; rely on retries/backoff, don't flip to polling loops. 6. Subscribe to `CONNECTION_UNHEALTHY` / `WEBHOOK_UNHEALTHY` notifications to catch failures early. ## References - Introduction to webhooks (payload, signatures): https://docs.unified.to/reference/webhooks - Virtual vs native webhooks: https://docs.unified.to/concepts/virtual_webhooks - Create & configure webhooks: https://docs.unified.to/guides/how_to_create_and_configure_webhooks - Filter webhook events: https://docs.unified.to/guides/how_to_filter_webhook_events - Troubleshoot unhealthy connections: https://docs.unified.to/guides/how_to_troubleshoot_unhealthy_connections - Troubleshoot unhealthy webhooks: https://docs.unified.to/guides/how_to_troubleshoot_unhealthy_webhooks - HTTP status codes: https://docs.unified.to/reference/rest