Solutions Lakehouse migration

Terabytes, reconciled.

Full-history migration from SAP and Oracle into Snowflake, Databricks, Microsoft Fabric or S3, with record counts and control totals proved for every table — then change-data capture keeps the lakehouse current, minutes behind the source. Lima Backbone is the SAP-aware ingestion layer that sits beside your lakehouse.

Two-week proof on your largest tables: terabytes moved, rows per minute, reconciliation variance, and cost per terabyte beside the metered alternative you run today.
SOURCE · SAP ECC / S/4HANA · ORACLEMARA · MARC2.9M rowsSTPO · STKO18.4M rowsEKKO · EKPO1.2M rowsVBAK · VBAP6.1M rowsBSEG1.2B rows · partitioned by periodLIMA BACKBONE · PARALLEL WORKERSw01w02w03w04w05w06w07w08w09w10w11w12w13w14w15w1632 → 128 vCPU · checkpoint restartjobs → task threads → query workersLAKEHOUSE · SNOWFLAKE / DATABRICKS / FABRIC / S3MARA · MARC2,914,220 = 2,914,220STPO · STKO18,406,113 = 18,406,113EKKO · EKPO1,204,881 = 1,204,878 · 3 to ownersVBAK · VBAP6,102,447 = 6,102,447BSEG1,204,118,930 = 1,204,118,930THEN CHANGE DATA · MINUTES BEHIND · DELETES HANDLED
Tables extracted in parallel, landed, and checked one by one. A variance becomes an exception with an owner; the rest is proved. Then change data takes over.
~30 min
220M rows · 615 tables · ~30 GB end to end on 64 vCPU
80 GB/day
moved in production on the engine
32 → 128
vCPU: the cluster grows for the initial load and shrinks after it
1m 48s
change-data lag behind SAP once the initial load is done
showcase estate
Per table
record counts and control totals compared and signed
product
2 weeks
Terabyte Proof-of-Scale on your own largest table set
offer
How a migration runs

Profile, load, prove, then keep it current

Profile and size week 1Every source table inventoried with row counts and sizes; large tables get a partition plan (by fiscal period, plant or document range); dependencies ordered so master data lands before transactions.
Initial load in parallel week 1–2Full-table extraction on open SAP interfaces, partitioned and run across the worker pool; a 1.2-billion-row table becomes a set of monthly partitions that load side by side. A failed partition reruns alone from its checkpoint.
Reconcile every table as each landsRecord counts and control totals compared with the source; any variance becomes an exception with an owner and a reason. The report is signed when every table matches.
Switch to change data week 2Inserts, updates and deletes captured from timestamps, change pointers or triggers keep the lakehouse minutes behind the source; deletes are applied, late-arriving rows replayed from the watermark.
Cut over week 2Reporting, analytics and agents read the lakehouse; the old extraction jobs are switched off with the reconciliation report as evidence.
Parallelism, plainly

How the engine moves a terabyte

A migration job splits into task threads, and each thread runs query workers against a partition of a table. The cluster grows from 32 to 128 vCPU for the initial load and shrinks again afterward, so the capacity you pay for follows the work.

Job · accounting historyBSEG · BKPF · 3 tablesJob · sales historyVBAK · VBAP · LIPS · VBRPJob · master dataMARA · MARC · KNA1 · 40 tablesJOBS · ONE PER TABLE GROUPTHREADS · ONE PER TABLEThread · BSEG1.2 billion rowsThread · BKPF180 million rowsWORKERS · ONE PER PARTITION (BSEG BY MONTH)2024-01✓2024-02✓2024-03✓2024-04✓2024-05✓2024-06✓2024-07✓2024-08rerun2024-09✓2024-10✓2024-11✓2024-12✓… 132 monthly partitions in all; each one checkpoints;a failed one reruns alone (2024-08) while the rest carry on.The cluster grows for the load and shrinks after it32 → 128 vCPU · read limits and time windows keep the ERP responsive · licensed by capacityReconciled per partition, then per tablecounts and control totals per month against the source · the table reads Reconciled when every partition does

Jobs → task threads → query workers

One job per table group, threads per table, workers per partition. The scheduler keeps every core busy and the source system inside the load limits your Basis team sets.

Recoverable by design

Every partition checkpoints. A network drop or a source timeout reruns one partition, and the rest of the load carries on.

Source-friendly

Read limits, time windows and throttling per source keep the ERP responsive during the day; the heavy partitions run in the window you choose.

Elastic, licensed by capacity

The tier sets the ceiling; there is no charge per row or per table moved, so a full-history migration costs the same as a quiet month at the same tier.

The reconciliation report

What signed-off looks like

Structure of the report Lima produces for every migration; the figures are illustrative.

TableSource rowsLoaded rowsVarianceControl totalStatus
MARA · MARC2,914,2202,914,2200—Reconciled
STPO · STKO18,406,11318,406,1130qty 1.20 × 10⁹Reconciled
EKKO · EKPO1,204,8811,204,8783 → ownersnet value matched3 exceptions
VBAK · VBAP6,102,4476,102,4470qty matchedReconciled
BSEG partitioned by period · 132 partitions1,204,118,9301,204,118,9300amount matched per periodReconciled
Illustrative. Every table gets a row; every variance gets an owner; the report is signed when the last row reads Reconciled.
Landing formats

Native tables in each platform, open formats on storage

TargetLands asStatus
SnowflakeNative Snowflake tables, loaded in parallel and registered in your catalogShipped
DatabricksDelta tables in Unity CatalogShipped
Microsoft FabricFabric Lakehouse tables in OneLakeShipped
S3 and object storageParquet, partitioned, readable by any query engineShipped
Iceberg and Delta on object storageOpen table formats written directly to storage, for engines that read them without a platform in betweenScheduled · standards plan
Statuses match the published standards conformance page; the open-table-format line is on the schedule and dated there.
Advantages, laid out

Counts proved, per table

The signed reconciliation report replaces the spot checks most migrations rely on.

product
Open SAP interfaces, ECC-first

Table reads, CDS views and change-data capture, which SAP keeps open; the same pipelines carry across the S/4HANA move.

product
No charge per row

Capacity-based subscription, so a 4 TB history costs the same as a nightly delta at the same tier; your lakehouse compute stays for analytics.

pricing principles
Minutes behind after the load

Change-data capture with deletes applied and late rows replayed keeps the lakehouse current.

showcase estate · 1m 48s
The same engine for the next job

Planning loads, write-back to the ERP, archiving and replication run on the same platform afterward.

product
Your data stays yours

Native tables in your platform and Parquet on your storage; readable without Lima from day one.

product
Lifecycle

A migration uses four stages at full volume; Return is available the day you need planning write-back.

One engine underneath every page in this family. It reads from any source and writes to any target, runs batch loads or change-data streams, handles a nightly delta or a multi-terabyte migration, and is licensed by capacity, so the bill stays the same however many rows move. Lima's fabric answers questions on data in place; Lima Backbone moves, transforms and returns data on a deadline. When to federate and when to pipeline · The Backbone hub.

Terms on this page

Plain definitions

LakehouseA data platform that stores tables on object storage in open formats and queries them with warehouse-style SQL — Snowflake, Databricks, Fabric and similar.
PartitionA slice of a large table, by month or plant for example, that loads on its own and reruns on its own if it fails.
WatermarkThe last change position a pipeline has processed; late rows are replayed from it.
Control totalA sum (amount, quantity) computed on both sides and compared, alongside record counts.
Iceberg / DeltaOpen table formats that add transactions and schema history to Parquet files on object storage.
CheckpointA saved position in a load that a rerun starts from, so a failure costs one partition and nothing more.
Questions we hear

Asked before the first load.

How big can the initial load be?

The largest documented run is about 220 million rows across 615 tables in roughly 30 minutes on 64 vCPU (estimate), with 80 GB a day moved in production. The Terabyte Proof-of-Scale runs on your own largest tables so the number you see is yours.

Does the ERP slow down during the load?

Read limits, time windows and throttling are set per source. The heavy partitions run in the window your Basis team approves.

What happens if a table fails to reconcile?

The variance becomes an exception with an owner and a reason. The affected partition reruns from its checkpoint, and the report stays open until every table matches.

Can we keep our lakehouse's own ingestion for other sources?

Yes. Lima is the SAP-aware ingestion layer beside the lakehouse; other sources can continue as they are, and the fabric reads the lakehouse live either way.

Which change-data method is used on ECC?

Timestamps, change pointers or triggers, depending on the table. Log-based capture on the HANA database by third-party tools is restricted by SAP, so Lima uses application-level methods.

What about the ODP RFC deadline?

Lima's extraction methods sit outside the ODP RFC interface, so the migration is unaffected by the December 2026 change. The exit page explains the note.

Run the Terabyte Proof-of-Scale

Two weeks on your largest table set: terabytes moved, rows per minute, reconciliation variance, and cost per terabyte beside the metered alternative you run today.