Passing data to a webhook automation for scripting

The webhook trigger for Airtable automations is a really useful tool. However, there are some issues that have been plaguing users ever since it was added.

Problem #1: access to primitive values only

The current behavior of the webhook trigger only allows access to primitive-value properties in a nested structure to be passed to later actions.

Here’s a quick overview of what I mean. If you’re new to scripting, an object in JavaScript is a structure that stores key-value pairs. Each key is also referred to as a property on the object. Here’s a brief sample:

{
    firstName: "Justin",
    lastName: "Barrett
}

This object has two properties—firstName and lastName—with the value associated with each property being a literal string.

Sometimes the value assigned to an object property will be another object, as in this example:

{
    firstName: "Justin",
    lastName: "Barrett,
    address: {
        street: "123 Main Street",
        city: "Sometown",
        state: "NV",
        zip: "12345-6789"
    }
}

Most developers would like to traverse the entire structure freely. However, as I said above, Airtable’s webhook automation trigger only allows access to primitive-value properties, meaning properties with values that are primitive types: string, number, boolean, etc.

In this latest example, only firstName, lastName, and the properties of the address object would be considered primitive-value properties. The address object itself isn’t directly available because its value is an object, which is not a primitive type.

For uses that don’t involve scripting, this generally isn’t an issue. However, it’s a pain if you want to use the incoming data in a script action. Instead of passing the entire address object to the script, you have to pass its properties one by one. The more nesting exists in the incoming structure, the more of a pain it is to use.

Problem #2: structure changes require repeated testing

To make anything from the webhook trigger available in a script, you have to test the trigger, which bakes in the structure of the incoming data. While this is useful in theory, it’s cumbersome in practice if the data structure is evolving during development.

Looking at the above samples, if I had passed the first object in my first test to the trigger, then realized later that I had to include the address data as well, I’d have to test the trigger again with the revised data structure before anything from that structure could be used. The more the structure is revised, the more re-testing must be done. As available properties change on the incoming data, input variables might also need to be changed.

My Solution

The solution that I’ve been using for a while now to work around both of these problems is to take the actual data structure that I want to pass and nest it inside of a single root property as a JSON string.

When calling any endpoint with JSON data, the setup typically looks like this (using my second sample above as an example):

const options = {
    method: "POST",
    headers: {"Content-type": "application/json"},
    body: JSON.stringify({
        firstName: "Justin",
        lastName: "Barrett,
        address: {
            street: "123 Main Street",
            city: "Sometown",
            state: "NV",
            zip: "12345-6789"
        }
    })
}

If this were being passed to an Airtable webhook automation, only those primitive values mentioned above would be accessible.

Here’s the same thing with a single data property at the root, with its value being the stringified version of the data object that I actually care about:

const options = {
    method: "POST",
    headers: {"Content-type": "application/json"},
    body: JSON.stringify({
        data: JSON.stringify({
            firstName: "Justin",
            lastName: "Barrett,
            address: {
                street: "123 Main Street",
                city: "Sometown",
                state: "NV",
                zip: "12345-6789"
            }
        })
    })
}

On the receiving end, the webhook trigger only provides access to the data property, which can then be extracted inside a script action after adding it as an input variable:

const { data } = input.config()
const { firstName, lastName, address } = JSON.parse(data)

If the core data structure ever changes, I don’t need to re-test the webhook trigger because data is still going to be the only property that the trigger needs to know about. I also don’t need to update any input variables for the script action. I only need to update the script itself to extract the appropriate property values from data.

While this solution might not work in all use cases, I hope that it will help when circumstances permit.

4 Likes

Thanks @Justin_Barrett for laying it out. I was not aware of the limitation with access to nested values. That is a good hack to run the second .stringify()!

This is a great technique when you have control over the payload, like when calling a webhook in one base from a script in another base.

Things get a little harder when the data is coming from a source that you do not control. As far as I can tell, Make.com can convert incoming data from a webhook into stringified JSON, but it cannot access the entire payload as a single entity either. Make.com must know top level keys, so you cannot have a generic scenario that can easily handle top level changes in the payload shape. (@ScottWorld please correct me if I’m wrong!)

Does anyone know of a no-code/low-code integration service that could JSON.stringify() the entire payload of an incoming webhook, regardless of changes in payload shape? Would autocode do this? (@bfrench I vaguely recalling having a similar conversation with you on this topic.)

Definitely, though their free plan has become a lot more limited after they changed their plan structures last month.

1 Like

Unless you were able to fully script the enumeration of the JSON object.

This is possible in Airtable script (of course) but lacking access to the JSON payload itself, exposes just how deeply flawed the webhooks integration architecture is. And yes - we did discuss this at length last year, perhaps even the year before.

Any NodeJS service is able to do this.

1 Like

You could also use https://pipedream.com/ for that. It is similar to Zapier in the workflow layout, but allows to run Node code blocks.

And the middle code block runs JSON.stringify()

The Node block allows also for NPM packages, so it can be interesting simpler alternative to Lambda/Cloud functions.

2 Likes

Yes, I believe you are correct, @Kuovonne, although I haven’t tested this myself.

Make can automatically recognize top level structure changes with their Parse JSON module, but it seems like the problem is that for subsequent modules, you would still need to specify “by name” which keys you want to pull data from.

For example, the screenshot below shows the Parse JSON module — you can either specify the structure in advance, or if you leave it blank, Make will make its own best attempt to parse the JSON.

But I think you will still need to be manually involved to reconfigure the future modules in the scenario.

And therein lies the rub.

1 Like

Bingo! And therein lay the cement poured without any thought by the Airtable engineers; apparently, they have a “hard” time seeing the bigger picture. In vastly most cases, you don’t have control of both the sending and receiving format; you are forced to proxy this flow to add the missing stringified payload.