Two-Way Real-Time Sync: Not Lovin' It

Is it me? Or is this new expansion of a lousy idea likely to lead to more issues?

Real-Time?

Nothing says “real-time” like a polling process that also comes with menu options to override it.

As I understand it, sync has a five-minute latency because it is polling for changes. By definition, this is not real-time. It may be suitable for many business applications, but it should be called what it really is.

Sync is Copying

There’s no other way to explain what sync does except make a duplicate of your data. Duplication of any data is a bad idea. This is the stark difference between using data by reference vs value. Leaning on data values that are copied is architecturally unsound. Why? Latency.

Latency is antithetical to real-time, and real-time is needed to avoid being misled by your data.

Latency is just the tip of the rolling iceberg. Imagine Base (x) syncs to base (y), and then base (z) uses an API call into base (a) for data that is about to change in base (y). Now you have layers of dependencies that are at risk in new ways. e.g., what if one of the sync processes is disrupted?

Events and Messaging

In 2018 I asked Airtable engineers if sockets-level integration was on the roadmap. Crickets. I asked again in 2019, 2020, and 2021. I stopped asking because they’re apparently not considering a modern communications layer between Airtable bases and external systems or other internal bases.

As early as 2018, I wanted to wire up a base with real-time motor-vehicle accident events directly from first responder agencies. This never happened, of course. Events didn’t make a debut for another three years, and there’s still no communications infrastructure that would support real-time messaging.

Polling

Much has been said about the massive polling cesspool pushed upon Airtable instances through the API by the likes of Make, Zapier, and other adhesives platforms designed mostly for immediate gratification at the cost of relentless performance hits.

Upwards of 99% of all API calls in a polling architecture are wasteful. They draw energy from container-based Airtable instances with fixed compute stack resources. So, what does Airtable do? Double down on this risky idea, of course.

If the Interwebs were entirely based on polling, it would have collapsed 20 years ago. We got smarter, though, and realized polling models should be voided at almost all costs.

Airtable appears to be adding technical debt that could result in an overall performance crisis depending on how deeply makers latch on to this [seemingly] perfect feature that everyone has been asking for.

Am I being irrational? Is this yet another immediate gratification feature that will eventually require repayment with interest and principal?

1 Like

Good points as usual, Bill. I can see value in two-way sync but hadn’t really thought about the technical debt associated with the way Airtable is implementing it. And your note about real time motor vehicle accident reporting is something that’s been on my radar screen too for a while now but of course the entire process ends up being mired in politics and right-to-know law. I could go on, but perhaps that’s a personal conversation we could have at another time. Look me up if you are ever back in NH and I’ll buy you lunch!

1 Like

First-responder agencies need data to move quickly, so my work has sidestepped the political and privacy challenges. A motor vehicle accident, after all, is a public event. The medical needs of anyone injured are HIPAA issues. MVH is just one example - Airtable should support near-zero latency pipelines for data if not for enterprises, for everyone who cares to pay for zero-latency network services.

Ha! Where are you located?

1 Like