Introducing Airlift

External Apps
1

Hello Airtable Users! :wave:

I would like to introduce Airlift!

A free & open-source tool to automate uploading & merging *.csv or *.json data files with attachments to Airtable! Yes, you have heard correctly. With attachments! While not officially support by Airtable APIs, we have found an intuitive way to upload attachments to Airtable.

You can send and transfer data automatically without needing subscription platforms like Make or Zapier.

I hope Airtable users can benefit from using Airlift!

Love to hear your feedbacks! We are open to adding your requested features if it is within the scope of our tool. :grinning:

2 Likes
2

Hello V.R.,
That is very very amazing. Thank you for contribution and I am sure this is gone be useful.

And if there was a way to do reverse process, that would be even more deserving appreciation. Right now I resolve to the over-the-top Zapier workflow to bulk-download attachments in a record. It nags me Airtable doesn’t want users to be able to export attachments painlessly.

Is it difficult to do the reverse process with Airlift?

I am not a programmer so unless there is a video, I would lose attention quickly. The mapping is dome in a separate file?
–attachment-columns-map

Sorry, I guess this is way over my head if I ask basic questions like this.

3

Sorry for double post, decided to break my answer into two separate posts so that it’s not confusing or too long because it unrelated to the first post.

Regarding Airtable CSV import… it works mostly Okay but I do have some issues which are related to the way I use CSV Extension and the way I use it is just to import text, no attachments in my case.

  • The mapping is not permanent, if I have mapping pre-set, and I end up importing a wrong CSV, the mapping will be reset and Airtable import extension will try again guess the mapping pairs, and if I cancel the import process and then open the correct CSV to import, the mapping is broken with some pairs so I have to remap. For that reason I had to use multiple copies (I have about 10 copies of the CSV Import Extension in Airtable) to be able to pre-set mapping, plus another 10 copies as backup so I can restore the mapping if things get messed up.

  • Airtable CSV import extension only auto-maps a CSV field ones, I may have 2 fields in my Airtable, one being the matching field and another being the edit field (I do that because the edit field may be empty but the matching field is autogenerated in case the edit field is empty, this allows me to avoid imports into wrong records). Anyhow, the 2nd pair being identical must be setup manually. Because Airtable CSV Extension things that is not a good idea, it always overrides my mapping and I have to manually pair that 2nd pair before every import. That kind of relates to point above.

  • Airtable CSV Import extension doesn’t warn me if a field mapping is missing (again related to previous points)

  • Extension doesn’t warn me if I am importing the wrong file (maybe if I could set in the extension to warn me if the file name of the imported CSV does not match in part the preset file name when I set up the CSV import template. That would prevent the issues described above in many cases.

  • I have 100’s of fields so going through that longe list of Airtable fields when mapping it to my CSV is very tedious, the fields are not even sorted alphabetically, but rather by date when the field was created (this is one of the weird nags in Airtable). So I wish there was easier way to find the field i am looking for from the list of my fields. An ideal wish would be also if the fields that are paired, would go to the top of that list, see how the field order works in Interfaces, there selected fields move to the top, the bottom has list of hidden fields. Something like that could be nice to have also for that Airtable CSV extension.

  • This is just wish list… The extension could also give me better to preview what is being updated through the import of the CSV record, that would be nice to have.

  • Some CSV files have headers and other messy comments so I am dependent on using Easy Data Transfer app which is an amazing app although quite costly, to manipulate CSV data before import. I couldn’t imagine Airtable extension would be able to handle something like that, but a few basics would be nice, such as setting from which row the data is imported, which row is the header, which rows at the bottom to ignore, and maybe using something like a text or regex filter to ignore certain rows… again this is just a wishful thinking.

4

Regarding Airtable CSV import… it works

Great to hear that you manage to get it working at your end!

The mapping is not permanent, if I have mapping pre-set, and I end up importing a wrong CSV, the mapping will be reset and Airtable import extension will try again guess the mapping pairs, and if I cancel the import process and then open the correct CSV to import, the mapping is broken with some pairs so I have to remap. For that reason I had to use multiple copies (I have about 10 copies of the CSV Import Extension in Airtable) to be able to pre-set mapping, plus another 10 copies as backup so I can restore the mapping if things get messed up.

Could you provide a video example?

Airtable CSV import extension only auto-maps a CSV field ones, I may have 2 fields in my Airtable, one being the matching field and another being the edit field (I do that because the edit field may be empty but the matching field is autogenerated in case the edit field is empty, this allows me to avoid imports into wrong records). Anyhow, the 2nd pair being identical must be setup manually. Because Airtable CSV Extension things that is not a good idea, it always overrides my mapping and I have to manually pair that 2nd pair before every import. That kind of relates to point above.

Mapping to 2 or more columns could be possible in future version. We can look into it.

Airtable CSV Import extension doesn’t warn me if a field mapping is missing (again related to previous points)

Extension doesn’t warn me if I am importing the wrong file (maybe if I could set in the extension to warn me if the file name of the imported CSV does not match in part the preset file name when I set up the CSV import template. That would prevent the issues described above in many cases.

We plan to improve and add more validation options in future builds.

I have 100’s of fields so going through that longe list of Airtable fields when mapping it to my CSV is very tedious, the fields are not even sorted alphabetically, but rather by date when the field was created (this is one of the weird nags in Airtable). So I wish there was easier way to find the field i am looking for from the list of my fields. An ideal wish would be also if the fields that are paired, would go to the top of that list, see how the field order works in Interfaces, there selected fields move to the top, the bottom has list of hidden fields. Something like that could be nice to have also for that Airtable CSV extension.

Our tool is not a GUI tool. It is to be run on a terminal.

Is it difficult to do the reverse process with Airlift?

Not the present focus of our tool.

I am not a programmer so unless there is a video, I would lose attention quickly. The mapping is dome in a separate file?
–attachment-columns-map

Nope. All done in the same line of arguments.

There are examples shown in the GitHub repo.

Thanks for the suggestion!

1 Like
5

The very essence of this problem domain suggests that either (i) users will need a visual approach to mapping or (ii) an entirely different approach to selective field mapping.

I can appreciate the benefit of a pure CLI approach, but in the current climate of increasing integration and data model complexity, this tool will struggle to attract adoption.

I think up a lot of bad ideas almost every day, but I’m pretty sure the next one is not one of them.

This problem is best addressed with generative AI, which has proven the ability to not only perform intelligent field mapping but also generate the code required to achieve precise ingestion. In that sense, it is a CLI approach that requires no GUI and is likely useful for developers and everyday users.

The underlying architecture of this solution is quite clever and implemented very well. Lean into an AI approach using your code as the learner shot. I think you will be surprised how powerful generative AI will prove if given the architectural answer.

6

The very essence of this problem domain suggests that either (i) users will need a visual approach to mapping or (ii) an entirely different approach to selective field mapping.

This problem is best addressed with generative AI, which has proven the ability to not only perform intelligent field mapping but also generate the code required to achieve precise ingestion. In that sense, it is a CLI approach that requires no GUI and is likely useful for developers and everyday users.

Lean into an AI approach using your code as the learner shot. I think you will be surprised how powerful generative AI will prove if given the architectural answer.

Hi there! if you are a developer, feel free to develop a front-end. :sweat_smile:

I can appreciate the benefit of a pure CLI approach, but in the current climate of increasing integration and data model complexity, this tool will struggle to attract adoption.

As mention, in our slack chat, this tool is not intended for business and organisation. It is targeted for individuals, small or local environments.

I think up a lot of bad ideas almost every day, but I’m pretty sure the next one is not one of them.

Great! Looking forward to your next solution!

7

Ha ha! Yeah, I’ll get right on that after I solve 250ms latency at 99.9999 accuracy for generative AI in one of the largest drive-thru’s on the planet. I have time to make observations. No time to fix everything I observe.

8

Well, let’s welcome @V.R for introducing a generous contribution to the community. While not quintessential, as not everybody has time at hand and contract dollars to devote to 1000’s of hours for free, with effort of smart people, the work can be shared, that is called open source, I think, forgive me my uneducated ignorance because I am not a programmer just a humble freeloader making a bare living in a hard way. For comparison, there is not much going on with Airtable other than price increase so far and talent has been drained so I don’t expect surprises in the coming few months. If using Airlift as an example maybe discouraging to non-coders like me, perhaps it will encourage others who are able to contribute and encourage more to join. AI is useful, yes, but it makes the most frustrating mistakes. Sometimes I feel I would to kick AI in his pants or in her skirt, or whatever pronoun it prefers to use.
The Airlift tool is specifically designed for the poor dogs. That’s why no AI and no GUI. And no desert either.

9

I will try to do the video, but won’t bee soon, because I have almost 500 fields in the table, so it is a bit of a circus to do this. I’ll do it when I add a new CSV import template.

10

The AI I use to post in this forum, what a bugger, forgot to mention there are 2 subscription services.

Seems they haven’t switched to AI yet either, so there we go , it’s a million dollar opportunity right here at @bfrench your hands :money_mouth_face:

11

It does, but typically, this reflects the prompter and the approach used, not the LLM. If you’ve rarely or never experienced deterministic outcomes, you need to work harder at generative AI. Maybe that’s too big of a reach if the payoff isn’t financially practical for @V.R . But I sense from your frustration and feedback that either a GUI is needed (I think you mentioned three cases where it would help), or there needs to be more feedback from the CL tool especially when there are hundreds of fields.

Indeed. It is possible to overcome the issues you raised with generative AI. I’m suggesting it only because it is a great tool with great promise for the community, not only “poor dogs”. :wink:

12

Are you suggesting @V.R could achieve the goal faster by using A.I.? So in other words, by using AI, the solution can be achieved in what… fraction of time / effort that it takes doing the way it is now provided?

13

That’s not really the right question to be asking.

I think the way to approach this is to accept the reality that almost every conversion integration is unique. That said, no single code template, even with variant capabilities, can predict the possibilities required. Furthermore, there are cases where the number of fields and data types require a lot of mapping.

This problem domain is littered with attempts by some great developers to unify an import strategy based on a codified approach. From CSVs to SQL, and even this clever attempt to embrace attachments. Almost all ingest models have limits that create headaches. The edges are always rough and the success rates are low. The work is tedious, and often never completed with any degree of satisfaction.

So what could possibly overcome all these challenges and even do it better? I can only think of one - generative AI that can consider all source data types and fields, map the fields automatically into the target schema, suggest schema changes, produce the code, and run it.

I believe this is the pathway to an intelligent solution that everyone could benefit from and one that may be able to address edge cases you could never anticipate or code for.

14

I can tell you now that my CSV pairing makes no sense to anybody except for me. The field names are cryptic. I use things like H_Pc_AfPtLsCoaRskSgFf_BfCmp_CyUs

This is one of 500 fields I have in the table. Now tell me how on earth will AI figure it out, when I can’t figure it out sometime myself, … well I do know what it is because I use a system of abbreviations, but the 2 or 3 letter word abbreviations are random and they mean to show some of the components if the formula in the field name.

I do it because I want to see it in the field name without having to showelling my mouse always about the bubble. Secondly it helps me with filtering fields in the filed picker when I look for a certain field.

Yes I could create another field just for CSV import and name it in a friendly way, to match the CSV file I am importing and that is what the CSV Airtable extension already does. But I have mere dozen of available fields in several of my tables, maxing at 500 almost.

And since Airtable doesn’t allow external apps see the Airtable formula (not even AI, as you mentioned earlier yourself), I can’t see how that would be more meaningful if there is a number and I have 100’s of fields that look alike.

AI can help if it reads text so it figures out what type of field it is and what is the meaning of it.

But if there is a field like the one example above, and I have 100’s of these kind of fields…

Yes, I could use another table, but I am already maxed out on everything possible in Airtable, I don’t have empty slots of anything much unless I go with Enterprise which I obviously can’t afford to pay even for one day :wink:

So my feeling is, future is AI, for sure, but right now I think at this level of maturity of Airtable, all AI would do for my CSV mapping is a luxus toy but not useful unless I babysit it and spend more time fixing it than doing mappings the way they are done today.

I couldn’t trust Airlift to do my mapping, even if it had some kind of AI matching, unless, of course, Airtable provides access to formulas beyond the table data to see the logic in Airtable. Without that, I don’t know how it would make sense if Airtable is so introverted now.

15

That’s nutty, but I see why you’ve leaned into this insanity.

Isn’t that a signal that maybe you should rethink your strategy?

You might be able to train an AI to see the picture more vividly, but that seems like a stretch.

image

That’s not a complete picture of how a generative AI process would maybe be of assistance. While even the most advanced LLMs struggle with numbers, they do not struggle with recognizing data types. Mapping based solely on field name was not in my vision. Rather, sample rows of data - even just one row - is enough to enlighten the AI with intelligence concerning mapping precision.

Tesla’s FSD is designed for a world where eyes see signage and features. Generative AI is designed for a world where words have meaning. You have managed to create a world based on cryptic tokens.

You may be right. Airtable has historically been a closed system. It’s why you can’t get good results building formulas with generative AI. Airtable knows this, and they intend to use this as a financial lure to use their AI at their chosen cost.

You, specifically, are in a corner. @V.R could use generative AI to his advantage, possibly making a lot of users happier.

16

Airlift 1.0.1 is released.

You can now map/copy value of one column to multiple other columns.

See example in the GitHub repository for usage.


### 1.0.1

**🎉 Released:**
- 22nd November 2023

**🔨 Improvements:**
- Added `--columns-copy` : Copys value of one column to multiple other columns (#25)
- Improved error handling logic when attachments are not found

Demo

You can now map/copy value of one column to multiple other columns.

17

Airlift 1.0.3 is released.

You can now upload .csv or .json payload file with a different key column in Airtable.

See example in the GitHub repository for usage.

### 1.0.3

**🎉 Released:**
- 13th December 2023

**🔨 Improvements:**
- Added `--rename-key-column` : rename the key column in the file to a different key column in Airtable (#27)
18

I don’t have a video, not much video to do, but here is example, I was importing a CSV, which I always do import, the CSV format changes slightly in the way that there is one less column in the updated version of the CSV.

The CSV import didn’t work as expected using the Airtable CSV import because the auto-mapping mixed up 2 key columns, one of them is the primary column for me.

The new CSV changed, however, not these columns. Some other previously mapped column (Expenses column) was removed from the new version of the CSV I am importing.

That seems confuses the CSV import and now some mapping broke.

Previously I was importing mapped:

Order Number → Order Number
Sales Record Number → Sales Record Number

Importing the new CSV now and it mixed up these two.

The rest of the columns seems imported correctly.

Unfortunate thing with the CSV import extension in Airtable is it doesn’t warn you if things are not normal:

a) the CSV template has changed
b) mapping is broken in some way

Both things could be accomplished easily.

So I am adding this here because you asked me for an example.

Maybe a good soul can wring a GUI for the Airlift.

Or maybe ChatGPT will get so good one way I can ask it to create a GUI :wink:

The issue with Airtable doesn’t happen very often, but when it does it happens in the most inconvenient of moments. Maybe I need to look at the online CSV import tools for Airtable although some of them do charge by line, regardless whether the CSV contains line of data or line of comments which I am not importing.

I am using application called Easy Data Transform and that using the transformation Stack and then Transpose and Compare Cols etc I compare 2 CSVs , the original template and the imported CSV to see if they match. It tells me if they are not matching. It’s a workaround as Easy Data Transform doesn’t have native CSV comparison, but it works via the Stack function.
However, I didn’t use the Easy Data Transform today, Forgot to use it. So went ahead and imported it to Airtable via Extension and it mixed up the columns because there was a small unrelated change in the CSV:

Here is the mix-up:

mapping
mapping726Ă—235 11.8 KB

19

I am having a second thought about this…

I don’t know… but… could I use this with Apple Script and using Automator to create a script to get this working?

If it is doable, I’ll try to get ChatGPT to create the script for me. If that works via applescript that would be awesome solution. Easy. I would just drop the CSV on the icon on the Dock on MacOS and then the Applescript would run and send the CSV Correctly mapped to Airtable!

Fabulous! Wondering if this would work.