Integration & reliability
Your data flows should make sense to the people who rely on them.
It started with…
CRM and ERP systems describe the same business differently. When the connection lives in an opaque script, a failed sync quickly becomes a call to its author.
So we built…
An Airtable control room for the sync engine. Operations can inspect jobs and run history; developers can review the underlying mappings. Both use the same configuration.
Technologies & official documentation
Take a look inside
Screen 1 of 6Operations view
Resolved names explain source, destination and write policy without hiding the mapping.
What we set out to change
Help teams understand what is synchronised, diagnose a failed run and adjust configured schedules without redeploying the engine.
How it is builtData model, code, controls and team ownership
The user journey
- 01
Define the job and its mapping in the shared reference.
- 02
Request or schedule execution through Airtable automation and the external worker.
- 03
Inspect the run, its metrics and errors before deciding what to retry.
Base structure
| Entity / table | Key fields | Relationships |
|---|---|---|
| Sync Jobs | Mapping JSON · schedule · enabled status · manual request | One job → many recorded runs |
| Sync Runs | Status · read / written / skipped counts · diagnostics | Each execution links to its job |
| Sync Types | Source and configuration references | Shared definitions classify jobs |
| Promotions | Matching keys · mapped destination fields | A common destination for operational reporting |
Technical implementation
- A TypeScript bridge runs the Python engine on Trigger.dev. Source adapters, mapping validation and destination writing are separate responsibilities.
- The REST writer resolves field identifiers, groups writes into batches of 10 and includes rate limiting.
- Preview/log-only support and read, written, skipped and error metrics make a run inspectable.
Engineering decision
Recompute the full series, write only changed cells
Adapted from the weekly-series writer. A complete recalculation can correct old periods, while a cell diff avoids rewriting unchanged values. Missing series are left untouched, an explicitly empty series clears previous counts, and dry-run is the default. This is repeatability for a normalised snapshot, not a distributed transaction or a concurrency lock.
def plan_updates(series_by_record, existing, field_maps): updates = [] for record_id, series in series_by_record.items(): desired = {} for metric, week_fields in field_maps.items(): values = series.get(metric) if values is None: continue # unavailable source: preserve this metric for week, field_id in week_fields.items(): desired[field_id] = int(values.get(week, 0)) current = existing.get(record_id, {}) changed = { field: value for field, value in desired.items() if int(current.get(field) or 0) != value } if changed: updates.append({"id": record_id, "fields": changed}) return updatesdef sync_series(series, existing, field_maps, batch_update, *, dry_run=True): updates = plan_updates(series, existing, field_maps) summary = { "records": len(updates), "cells": sum(len(row["fields"]) for row in updates), "written": 0, } if not dry_run and updates: # Injected adapter owns API batching and error reporting. summary["written"] = batch_update(updates) return summaryAdapted implementation excerpt
Controls and boundaries
- Credentials belong in the execution environment; a readable mapping does not need to contain their values.
- Retries, quotas and matching rules must be evaluated per source. This case does not claim exactly-once delivery.
Who can contribute
- Operations: inspect jobs, request runs and interpret execution history.
- Builders: maintain permitted mappings and schedules in Airtable.
- Developers: add adapters and test mapping or writer changes against the shared contract.
What supports this case
The live back-office was inspected in Human and Dev modes. The repository contains the mapping schema, engine and both released interface extensions. No synchronisation was triggered during review.