Most tables change a little at a time. Change capture reads the rows that were inserted, updated or deleted since the last read, and applies them to the copy in the order they happened. It works on SAP and Oracle, on other databases, on applications that push events, and on event streams.
A pipeline can copy everything, copy the rows past a marker, or read the system’s own record of changes.
The whole table is read and written to the target on every run. It is the right choice for a first load, for small reference tables, and for an archive extract, where the volume is moved once.
Each run reads only the rows whose marker column, such as a last-update timestamp or a sequence number, is past the value recorded at the end of the previous run. It is the right choice for large tables that gain rows and are refreshed a few times a day.
The system’s own record of changes is read: every insert, update and delete, with the values before and after. It is the right choice for tables that people watch during the day, such as orders, deliveries and stock positions.
| Full copy | Incremental copy | Change capture | |
|---|---|---|---|
| Freshness | As of the last run, usually nightly. | As of the last run, from hours to minutes apart. | Minutes behind the source. |
| Load on the source | Reads every row on every run. | Reads the changed rows. The marker column needs an index. | Reads the change record. The table itself is left alone. |
| What it needs | A read-only user, or the released reading interface of the ERP. | A reliable last-update column or sequence on the table. | Access to the system’s change record: SAP’s application-level paths, Oracle LogMiner, a database log or trigger, or an application that pushes events. |
| Deletes | Seen when the copy is rebuilt. | Seen when the source marks deleted rows. | Seen as DELETE events, with the removed row kept. |
The same five steps run for every source, from an SAP table to a Kafka stream.
Both ERPs keep a record of what changed. Backbone reads that record through the paths each vendor supports.
An ERP is the system that holds orders, stock, materials and finance. The change stream, the checks and the targets are the same for both ERPs. The paths used to read the changes differ, and the table sets them side by side.
| SAP ECC and S/4HANA | Oracle E-Business Suite (EBS) and Oracle ERP | |
|---|---|---|
| Paths used | Change pointers: SAP’s own record of master-data changes, such as a customer or a material. Table-level change capture: inserts, updates and deletes on a transaction table, with before and after values. Delta-enabled CDS views: released data views on S/4HANA that return only the rows changed since the last read. Extractor delta queues: the queues that SAP’s own extractors fill with changes. | LogMiner: Oracle’s own tool for reading the database’s change log. Backbone uses it where the database administrator allows it, and receives every insert, update and delete with before and after values. Last-update columns: where LogMiner is switched off, each run reads the rows whose last-update column is past the marker. |
| Other editions | SAP BW, SAP’s data warehouse, is read through its own connector. | Oracle Cloud is read through its released APIs and extracts. |
| Left out | The ODP-RFC replication interface and HANA database log capture. SAP Notes 3255746 and 2971304 restrict their use by third-party tools, and the ODP-RFC opt-out expires in December 2026. | Reads use a standard database user with read grants. The database keeps its standard setup. |
| Showcase estate | 2.8M events a day on the stream monitor, 42 ms p99. 8.2M events a day across all SAP paths. Lag 1m 48s, shown live. | 0.9M events a day through LogMiner, 210 ms p99. |
Beyond the ERP, changes arrive in four ways. Each lands on the same change stream, with the same marker, checkpoint and replay. An API is an interface a system publishes for other programs to call.
| Source | How the changes arrive | Timing | On the showcase estate |
|---|---|---|---|
| Webhooks | The source pushes each change to Backbone as it happens. | Seconds | The warehouse system pushes its DISPATCH, RECEIVE and TRANSFER events: 841K a day. |
| APIs | Backbone calls the source’s API at a set interval and reads what changed since the last call. | The interval you set | The 3PL connector polls and also receives pushes: 48K records a day. It was built with the connector kit (SDK) in 4 days and registered 72 endpoints on its own. |
| Event streams | Kafka, Kinesis and Azure Event Hubs are read in micro-batches: small groups of events processed together. | Seconds to minutes, as you set it per topic | Micro-batch interval set per topic. |
| Databases | The database change log is read where one is available. Otherwise a trigger records each change in a side table that Backbone reads. | Seconds to minutes | Snowflake through its own change streams: 0.6M events a day, 18 ms p99. |
One screen lists every source, its rate of change, and how long a change takes to pass through.
| Source | Method | Events / day | p99 |
|---|---|---|---|
| SAP S/4HANA | application change capture | 2.8M | 42 ms |
| Oracle ERP | LogMiner | 0.9M | 210 ms |
| Snowflake | stream + task | 0.6M | 18 ms |
| WMS | trigger + REST | 0.5M | 88 ms |
| 3PL | webhook (SDK connector) | 0.1M | 120 ms |
| Kind | Table | Column | Before → after |
|---|---|---|---|
| UPDATE | SAP.VBAP | KWMENG (ordered quantity) | old quantity → new quantity |
| INSERT | WMS.STOCK_ONHAND | new row | — → the new row |
| UPDATE | SAP.KNA1 | LAND1 (country) | United States → US |
| DELETE | Oracle.AR_INVOICES | removed row | the removed row → — |
| INSERT | 3PL.SHIPMENTS | new row | — → the new row |
p99: 99 of every 100 events passed through the stream within this time. Events per day is the count over the last 24 hours. The event feed shows the kind of change, the table, the column and the values before and after. All figures are from the showcase estate.
Two saved positions let a run resume after a hold with every change applied once.
A marker says where the last read stopped. A checkpoint says where the last write stopped. Between the two sit the events that have been read and are waiting to be applied, so a run always knows what is done and what is left.
On the showcase estate, a rule found 847 negative quantities in a warehouse feed at 02:14 and held three pipelines. The checkpoint was kept while the cause was traced to a new release of the warehouse system. At 02:58 the pipelines resumed from the checkpoint and replayed 841K events in the order they happened. At 03:14 the copy had caught up, with each change applied once, and the 06:30 deadline was met with 3h 32m to spare. The hold, minute by minute →
The right mode depends on how fresh the copy needs to be and on how the data is used.
| The need | Recommended mode | Why |
|---|---|---|
| A nightly report | Batch: a full or incremental copy on a schedule | The report is read once a day, so a copy taken after the day’s postings is fresh enough, and the load on the source stays low. |
| A planning run at 06:00 | Change capture during the day, then a batch load before the run | Changes are collected all day, so the load before the run is small and finishes well before 06:00. |
| A stock position for customer service | Change capture | Stock moves every minute. The person on the phone needs the position as of a few minutes ago, and change capture keeps the copy that close. |
| A partner feed | Webhooks or API polling, then an event-driven pipeline | The partner’s system pushes events, or is polled on a schedule. Each batch of events starts the pipeline that checks and forwards them. |
The lag of every source is measured, shown on the monitor, and compared with a limit you set.
Lag is the time between a change in the source and the moment the copy reflects it. Backbone measures it for every source from the timestamps on the events, and shows the current value and the p99 over the last 24 hours. On the showcase estate the SAP stream shows 1m 48s, and the Oracle stream shows 210 ms at p99.
For each source you set a limit. When the lag passes it, an alert goes to the channel you chose. On the showcase estate a data-quality anomaly reaches the data-operations channel on Slack in 340 ms, and a projected deadline miss of more than 15 minutes reaches PagerDuty. The same lag figure feeds the deadline clocks in orchestration, so a held stream is visible on the run dashboard as well. Deadlines and alerts →
| What is measured | Where it shows | What happens at the limit |
|---|---|---|
| Lag per sourcecurrent value and p99 | The change stream monitor, per source | An alert goes to the channel you chose: Slack, PagerDuty, Teams or ServiceNow |
| Events per dayper source | The monitor, last 24 hours | Shown for context beside the lag |
| Deliveryshare of events that reached the target | The monitor: 99.97% on the showcase estate | A missed event is replayed from the checkpoint |
| Time to the business deadline | The run dashboard in orchestration | PagerDuty is called when a miss of more than 15 minutes is projected |
Change capture reads the ERP’s own record of changes, so the transaction tables themselves are left alone. On SAP that record is the change pointers, the table-level change capture, the delta-enabled CDS views or the extractor delta queues. On Oracle it is the database change log, read through LogMiner. The read is small because it covers only the rows that changed.
On the showcase estate the SAP stream shows a lag of 1m 48s, and 99 of every 100 Oracle events pass through in 210 ms. Your own figure depends on the path and on the capture interval your team allows. It is measured and shown on the monitor from the first day.
The last checkpoint is kept. When the source is back, the run resumes from the checkpoint and reads the changes it missed in order, so each change is applied once. The lag rises during the outage, the alert fires when the lag passes the limit, and the lag falls back as the replay catches up.
Yes. Where the database administrator allows LogMiner, Backbone reads the database change log and receives every insert, update and delete with before and after values. Where LogMiner is switched off, Backbone reads by last-update columns at the interval you set. Oracle Cloud is read through its released APIs and extracts.
On SAP and Oracle the changes come from the application paths and from LogMiner or last-update columns, so triggers are unnecessary there. Triggers are one option for other databases that keep no readable change log: the trigger writes each change to a side table, and Backbone reads the side table.
Four weeks, one dataset, one target. On the showcase estate, SAP change capture was connected on day one and Oracle on day two, so the monitor showed their lag in the first week.
