Webhooks

This guide covers the basics of webhooks and their underlying mechanisms.

For our webhook API reference, see: Webhook API Overview

To learn how to use webhooks, see: How to create and configure webhooks

If you’re having trouble with webhooks, see: How to troubleshoot unhealthy webhooks

Background: What are webhooks?

Webhooks allow servers to send real-time notifications to other servers when a specific event occurs. They are sometimes called “HTTP callbacks” or “push APIs”.

Webhooks enable instant communication between different systems, making them crucial for maintaining data synchronization and triggering automated workflows. For instance, Stripe's payment processing system uses webhooks to notify a merchant's server immediately when a customer completes a purchase, allowing for instant order fulfillment and inventory updates.

Think of webhooks as a delivery service that brings packages directly to your doorstep. In contrast, traditional REST APIs are like repeatedly checking your mailbox to see if you've received any mail. With webhooks, you don't need to keep asking, "Is there any new mail?" Instead, the data comes to you as soon as it's available, saving time and resources.

Webhooks can make your synchronization strategy less complicated since your application will receive updated data directly, eliminating the need for frequent polling and reducing the risk of missing important updates.

Differences between polling for updates and receiving pushed updates

Polling for updates (via APIs)

The traditional way of polling for new data from an API is by calling a List<Object> endpoint with something like a updated_gte: Date filter - this returns data that has been updated since that date.

How often you poll for new data should be determined by how important that data is to your application as well as your budget. If the data isn’t time-sensitive i.e. you don’t need to have the data the moment it becomes available, then this strategy may be sufficient. But you need to take care of retry strategies when the API returns a rate limit error, which can make this method more complicated than using a webhook. You’ll also need to handle any network errors that could occur.

Pushed updates (via webhooks)

Modern APIs and applications leverage webhooks for synchronization because of their simplicity and timeliness. You, the app developer, do not have to create a retry mechanism to actively poll the server for new data or deal with rate-limiting and other network errors. The hard work is done by the third-party API provider (and us!) to send you updates as they come.

How do webhooks work?

Webhooks operate on a subscription model, where your application registers to receive updates about specific events from a third-party API provider or vendor. Here's a step-by-step breakdown of how webhooks typically work:

1. Create a webhook endpoint on your server that can receive POST requests. This endpoint will handle the incoming webhook data.
2. Subscribe to the events you want to monitor by registering your webhook endpoint's URL with the third-party API provider. This is usually done through the provider's developer dashboard or via their API.
3. Specify which types of events you want to receive notifications for. This could be anything from lead generation to payment confirmations to new deals, depending on the service.
4. Receive and process data. When a target event occurs, the provider sends a POST request to your registered endpoint. Your server receives this data in real-time and can process it according to your application's needs.
5. Handle the response. Your endpoint should respond promptly to the webhook request, typically with a 200 OK status, to acknowledge receipt. This helps prevent unnecessary retries from the provider.

By following this process, your application can receive and react to important events as they happen, without the need for constant polling. This real-time communication enables more efficient and responsive systems, allowing you to build more dynamic and interconnected applications.

Webhook payload

This is the payload that your server receives when webhook data comes in:

Values for type:

• INITIAL-PARTIAL: included with each page of results for the initial sync
• INITIAL-COMPLETE: included with last page (e.g. last 5 elements for the limit 100) of initial sync or with an empty list (no more results)
• VIRTUAL: included with every page when reading new data from a virtual webhook
• NATIVE: included with every page when reading new data from a native webhook

Best practices for signature validation:

The sig value is generated using the HMAC-SHA1 crypto method. The key is your workspace secret (found at app.unified.to/settings/api) while the contents are the combination of data and nonce i.e. HMAC-SHA1(workspace.secret, data + nonce)

Things to keep in mind:

1. Data: The data in the signature calculation refers to the data in the payload body as outlined in the table above.
2. Order of data: When validating the signature, the order of the payload data should match the order in which it was received.
3. Avoid adding extra spaces: When processing the payload, be cautious about introducing extra spaces. The original payload data is sent without extra spacing, and this should be maintained during validation.

In Python, when you deserialize and then re-serialize the payload data, the resulting string may not match the original input. This is because the default serialization process often adds extra spaces between fields and objects, altering the format of the data. The following code snippet demonstrates how to perform signature validation in Python:

def validate_webhook(request):
    r""" HMAC-SHA1(workspace.secret, data + nonce) """
    import hmac
    import hashlib
    import base64

    sig = request.data['sig']
    nonce = request.data['nonce']
    data = request.data['data']

    content = json.dumps(data, separators=(',', ':')) + str(nonce)

    key = bytes(settings.UNIFIED_WORKSPACE_SECRET, 'UTF-8')
    message = bytes(content, 'UTF-8')

    digester = hmac.new(key, message, hashlib.sha1)
    digest = digester.digest()

    decoded_signature = base64.urlsafe_b64decode(sig)

    return hmac.compare_digest(digest, decoded_signature)

The example below demonstrates how to perform validation in Javascript/Typescript:

import { createHmac } from 'crypto';

function validateWebhook(
    workspaceSecret: string,
    nonce: string,
    data: string | string[],
    sig: string
) {
    data = Array.isArray(data) ? JSON.stringify(data) : data;

    const originalSig = createHmac('sha1', workspaceSecret)
        .update(data)
        .update(String(nonce))
        .digest('base64');

    return originalSig === sig;
}

Native vs virtual webhooks

Most APIs do not support webhooks natively, but we've built a robust virtualization system that allows you to subscribe to “virtual” webhooks for some integrations exactly as if they were “native” webhooks.

Read more about the differences between native and virtual webhooks here: Understanding virtual webhooks.

Additional webhook features

At Unified.to, you can pull historical data from your connection (i.e. the third-party account), select which fields to receive data from, manually trigger webhooks for testing purposes, observe an audit trail of webhook activity, and monitor the health of your webhooks.

To learn how to use these features, see: How to troubleshoot unhealthy webhooks