Bulk and change-data replication from your systems of record into reporting databases, partner feeds, planning engines and lakehouses — insert, upsert, truncate-and-load or partition swap, on the schedule you set, with counts reconciled every run and every copy accounted for in the catalog.
Pick the pattern per target; the reconciliation is the same for all of them.
| Pattern | When it fits | How Lima runs it | What is checked |
|---|---|---|---|
| Truncate and load | Reference and master tables refreshed nightly; small enough to reload whole | Full read, atomic swap into the target so readers never see a half-loaded table | Row count and control totals against the source |
| Incremental append | Event and log tables that only grow | Rows since the last watermark appended; late arrivals replayed | Count of new rows; watermark advanced only after the check |
| Upsert by key | Master and transactional data that changes in place | Inserts and updates merged on the business key; deletes applied from the change log | Counts, plus a sample of keys compared both ways |
| Partition swap | Large fact tables organised by period or plant | A partition rebuilt beside the live one and swapped in | Partition counts and totals against the source period |
| Change-data stream | Copies that must stay minutes behind | Inserts, updates and deletes captured continuously; ordered replay after any pause | Event counts per interval; lag shown on the dashboard |
Cron-style, with run windows and deadlines attached, so a late run is visible before it matters.
A webhook from the warehouse system, a file arriving, a message on a queue.
When the upstream copy finishes and reconciles, the downstream one starts.
A gate that fails pauses the run and its deadline clock; the run resumes from its checkpoint once the fix lands.
Existing job schedulers trigger Lima pipelines and receive their status, so the operations team keeps one calendar.
SQL Server, PostgreSQL, Oracle and warehouses refreshed for dashboards and finance reports, with counts proved each run.
Net-change copies before the MRP run and the weekly S&OP cut-off.
Planning integration →3PLs, suppliers and customers receive scoped extracts through SFTP or API, with the access policy applied to every row before it leaves.
Golden-record corrections written back to the systems that hold them, each write acknowledged.
Deleted rows are captured from the change log or a full compare and applied to every target that holds them, so reports stop counting cancelled orders.
A row that arrives after its window is picked up from the watermark and applied in order, with the counts corrected.
A new column, a changed type or a dropped field is detected before the load and routed as an exception to the pipeline owner; targets are never broken silently.
A paused run resumes from its checkpoint and replays events in order; the deadline clock pauses with it.
Each copy is a catalogued asset with its freshness, subscribers and cost. When a copy is retired, its subscribers get notice and a migration path; when it changes, they see the lineage.
Each copy shows when it was last reconciled and who reads it; 84 subscribers were notified automatically during the showcase incident.
A field rename on a data product gave 32 consumers 14 days' notice and migrated them with zero breakage (showcase estate).
Compute per copy and per team is visible in the product, so a copy nobody reads is easy to find and retire.
One engine underneath every page in this family. It reads from any source and writes to any target, runs batch loads or change-data streams, handles a nightly delta or a multi-terabyte migration, and is licensed by capacity, so the bill stays the same however many rows move. Lima's fabric answers questions on data in place; Lima Backbone moves, transforms and returns data on a deadline. When to federate and when to pipeline · The Backbone hub.
Change-data copies ran 1 minute 48 seconds behind SAP in the showcase estate, with 8.2 million events a day. Bulk copies run on the schedule and window you set.
Yes. The access policy that filters a user's query filters a partner feed the same way, row by row and column by column, before the file or API response leaves.
The run pauses, keeps its checkpoint and its place in the change stream, and resumes when the target is back. Nothing is skipped and nothing is loaded twice.
No. Change-data capture from timestamps, change pointers and triggers is part of the platform; it uses interfaces SAP keeps open and needs no additional SAP licence.
Dependency triggers start a downstream copy only after the upstream one has reconciled, and the lineage shows which copy fed which.
Yes. Enterprise schedulers trigger Lima pipelines and receive their status; Lima's DAGs are standard Apache Airflow DAGs.
One source, one target, one pattern, reconciled every run — live in a week, then extend to the rest.
