A pipeline is drawn on a visual canvas, one step at a time. Every step writes its output to a table you can open, so you can check the rows before you add the next step. There are 150+ operators for the common work, and Python for anything else.
A pipeline moves data from one system to another and changes its shape on the way.
Every pipeline has the same three parts. It reads data from a source, such as an ERP table or a file. It changes the data: cleans it, joins it with other data, and calculates new columns. It writes the result to a target, such as a data warehouse or a planning application. The three parts are called extract, transform and load.
On the Backbone, every step writes its output to a table of its own before the next step reads it. While you build, you can open any of those tables and see its rows, the columns that came in, the columns that went out, and the record count. When you change a step, the tables after it are rebuilt from that point.
The product calls these linked tables. They are the reason a pipeline can be checked step by step on real data, and the reason one pipeline can reuse a table that another pipeline produced.
The same seven steps apply to a nightly copy and to a planning load.
Backbone reads an ERP through the interfaces the vendor supports. The pipeline canvas is the same for both.
An ERP is the system that holds orders, stock, materials and finance. The table sets SAP and Oracle side by side: how tables are read, how changed rows are read, which other editions are covered, and what is left out.
| SAP ECC and S/4HANA | Oracle E-Business Suite (EBS) and Oracle ERP | |
|---|---|---|
| How tables are read | Through released function modules, BAPIs and CDS views. A BAPI is a function SAP publishes for reading or writing one business object, such as an order. A CDS view is a database view that SAP defines and releases. | Through JDBC, the standard database connection. A large table is split into partitions so that several threads read it at the same time. |
| Incremental reads | Application-level change capture: change pointers, table-level change capture, delta-enabled CDS views and extractor delta queues. Each path returns the rows that changed since the last read. | By a last-update column on each table. Where the database administrator allows it, LogMiner change capture reads the changes from the database log. |
| Other editions | SAP BW, SAP’s data warehouse, is read through its own connector. | Oracle Cloud is read through its released APIs and extracts. An API is an interface a system publishes for other programs to call. |
| What is left out | The ODP-RFC replication interface and HANA database log capture are left out, because SAP Notes 3255746 and 2971304 restrict their use by third-party tools. | Reads use a standard database user with read grants. The database keeps its standard setup. |
An operator is one step on the canvas. The 150+ operators fall into four families.
| Family | What it does | Examples |
|---|---|---|
| Cleansing | Puts values into a consistent form before anything is built on them. | Trim and standardise text; fill or reject empty values; find duplicates, exact or fuzzy; convert types. |
| Enrichment | Adds columns from other data or from a calculation. | Lookup; derived columns; aggregation; pivot and unpivot, which turn rows into columns and back. |
| Validation | Checks each row against a rule and separates the rows that fail. | Format and pattern (regex); range; referential integrity, which checks that a key exists in the table it points to; business rules. |
| Advanced | Runs code or a model as a step. | Machine-learning model scoring; Python scripts; calls to a web service (REST API); SQL with window functions, which calculate across a set of related rows. |
Python Script Transformer. Write Python as a step on the canvas. It reads the table before it and writes the table after it, like any other operator.
Job Server. Run a program as a step: .py .java .scala .jar .R .sql .exe. The pipeline passes parameters in, and captures the errors and the output files.
How the data is read from the source, and how it is written to the target, are settings on each pipeline.
| Extract mode | What it does |
|---|---|
| Full | Reads the whole table or view. Used for a first load, for small reference tables and for archive extracts. |
| Incremental | Reads only the rows past a marker column, such as a last-update timestamp or a sequence number, and records the new marker at the end of the run. |
| Change capture | Reads every insert, update and delete from the source’s own record of changes, with the values before and after. Change capture → |
| Event-driven | Reads from Kafka, Kinesis or Azure Event Hubs in micro-batches: small groups of events processed together, at an interval set from seconds to minutes. |
| Load mode | What it does |
|---|---|
| Insert | Adds the rows to the target. |
| Upsert (merge) | Adds a row when its key is new, and updates the row when the key already exists. |
| Truncate and load | Empties the target and writes the full result. Used for small tables that are rebuilt on each run. |
| Partition load | Replaces one partition, such as a date, a plant or a company code, and leaves the rest of the table as it is. |
Every load is set up for its target. The batch size, the number of parallel writes and the file format are settings, chosen for the warehouse, lakehouse, database, file location or planning application the data goes to.
The three letters stand for extract, transform and load. The order of the letters says where the transform runs.
In ETL, the data is transformed on the Backbone’s own engine and the finished tables are loaded into the target. In ELT, the raw data is loaded first and the transform runs on the warehouse’s own compute. A mix does a light clean on the way in and the heavy transforms in the warehouse.
The pattern is a setting on each pipeline. The licence is sized by the capacity you run on, so all three patterns cost the same.
| Pattern | Where the transform runs | When to choose it | Example |
|---|---|---|---|
| ETL | On the Backbone engine, before the load. | The target has no compute of its own, such as a planning application, an API or a file. Or the transform is heavy, such as a bill-of-materials explosion. | SAP order, stock and material tables are transformed into the planning dataset and loaded into Kinaxis before the 06:00 planning run. |
| ELT | In the warehouse, after the raw data has landed. | The target is a warehouse with its own compute and the modelling team writes its transforms in SQL there. dbt, a tool such teams use for SQL transforms, runs in the same schedule. | Oracle EBS tables land raw in Snowflake every night, and the warehouse builds the reporting tables from them. |
| A mix | A light clean on the Backbone engine, the heavy transforms in the warehouse. | You want profiling, standard formats and exception records on the way in, and warehouse-side modelling after that. | Customer records from SAP and Oracle are standardised and de-duplicated on the way in; sales reporting is modelled in the warehouse. |
A template is a pipeline with the pattern, the operators and the load mode already chosen. Point it at your connection and your column mapping.
The tests run again on every change to the pipeline and on every run where the source columns changed.
A golden test is a saved input with a known correct output. When the pipeline changes, or when the columns of a source change, every golden test runs again and compares its output with the saved result. A test that fails names the column and the step.
On the showcase estate, five golden tests over 218K rows sit behind the certified view v_otif_daily. One night the SAP extract arrived with the column EDATU, the requested delivery date, under a new name. The sequence on the right is what happened.
The same test catches a changed column in an Oracle view. When a column in the Oracle extract is renamed or changes its type, the test that depends on it fails in the same way and the same hold applies.
A certified view is a table that has passed its tests and is marked as safe to use. Draft is the status before that.
EDATU, the requested delivery date, renamed. The input columns of the join step differ from the columns the tests were saved with.v_otif_daily loses its certified status. Every dashboard that reads it shows the status in place of a number.OTIF is a delivery measure. It counts the order lines a customer received on time and complete.
An order line counts as on time and in full when two things are true. It was dispatched on or before the requested delivery date. And the dispatched quantity equals the ordered quantity. OTIF % for a customer in a week is the number of such lines divided by all lines in that week.
On the showcase estate the rule marks a customer-week below 90% as an exception, so the supply-chain team sees which customers fell short and by how much. The pipeline runs every night on 841K order lines, finds 218 exception rows, and refreshes the certified view v_otif_daily. A supply-chain analyst built it in about two hours with the visual operators and 0 lines of SQL.
SAP.VBEP holds the requested delivery date (EDATU) and the ordered quantity (KWMENG) for each order line: 841K rows in the nightly run.WMS.DISPATCH holds the dispatch date and the quantity sent for each order line.v_otif_daily is the table the board report, the supply-chain report and the OTIF API read. Five golden tests over 218K rows guard it.The visual operators cover the common work: reading, joining, cleaning, calculating, checking and grouping. On the showcase estate a supply-chain analyst built the OTIF pipeline in about two hours with 0 lines of SQL, the query language of databases. SQL is available as an operator for the cases that need it.
Yes. The Python Script Transformer runs your Python as a step on the canvas; it reads the table before it and writes the table after it. The Job Server runs a whole program as a step, in Python, Java, Scala, R or SQL, with parameters passed in and errors and output files captured.
The canvas and the operators are the same. The reading method differs. SAP is read through released function modules, BAPIs, CDS views and application-level change capture. Oracle EBS is read through JDBC with parallel partitions, with last-update columns for incremental reads and LogMiner change capture where the database administrator allows it. Oracle Cloud is read through its released APIs and extracts.
Every column is profiled on the way in, and the golden tests run again whenever the columns of a source change. A test that fails names the column and the step, sets the certified view back to Draft and holds the batch. Once the mapping is corrected and the tests pass, the view is certified again and the run resumes from its checkpoint. This applies to SAP and Oracle sources alike.
In the Backbone’s own storage: a distributed file system and an in-memory layer. Extracted data lands there as it was read, and every operator writes its output table there. Lineage is recorded for each table, and the platform’s access rules apply to it.
Four weeks, one dataset, one target: your ERP into a data lake or a planning application, with a reconciliation report at the end.