Orchestration decides when each job starts and what happens next. A job can start in five ways, and each job has a deadline set on the business calendar. When a rule finds a data problem, the job pauses, keeps its place, and resumes from its last checkpoint once the data is fixed.
Orchestration is the part of the Backbone that runs the jobs: when each one starts, in what order, and what happens when something goes wrong.
A pipeline describes the work: read these tables, change them this way, write them there. Orchestration decides when that work runs. It starts each job at the right moment, runs the steps in the right order, and checks the result. When a step fails or the data is wrong, it decides what happens next and who is told.
Every job has a deadline. The deadline is the business event the job serves, such as the 06:00 planning run or the Friday review. While the job runs, Backbone projects the finish time and compares it with that deadline.
The steps on the right are a normal night on the showcase estate. The sections below take each part in turn: how a job starts, how its deadline is set, what happens when the data is wrong, how a flow is drawn, and how a run is watched.
A job starts on a schedule, on an event, after another job, when a quality rule releases it, or when your enterprise scheduler calls it. The counts are the showcase estate’s 48 pipelines.
| Way to start | What it means | Example | Showcase estate |
|---|---|---|---|
| On a schedule | A time expression sets when the job runs. It is written as a cron expression, the standard way to state a time such as every weekday at midnight or every 15 minutes, and shown with a readable label. | otif_nightly starts at 00:00 every day; finance_net_revenue at 02:00. | 28 pipelines |
| On an event or webhook | A message from another system starts the job the moment it arrives. A webhook is a message one system sends to another when something happens. | A dispatch message from the warehouse system starts the ingest at once. An SAP change event starts the enrichment that follows it. | 12 pipelines |
| After another job finishes | The job waits for the jobs it depends on and starts when all of them are complete. | The board report starts when v_otif_daily and v_net_revenue have both been refreshed. The month-end flow waits for every source pipeline. | 8 pipelines |
| Held or released by a quality rule | A quality rule holds the job while a batch of data is wrong and releases it when the batch is fixed. The job resumes from its checkpoint. | The three pipelines that read the warehouse feed paused at 02:14 and resumed at 02:58. | auto-pause |
| Started by your enterprise scheduler | Your scheduler keeps the company calendar, calls the Lima job, and receives the status back when it finishes. | The corporate scheduler starts the finance load at month end and records the completion status. | external |
The extraction steps from an ERP, the system that holds orders, stock, materials and finance, are ordinary tasks in a flow. An SAP extract, an Oracle extract and a file load can run in parallel and feed one transform step, which starts when all three have finished. A schedule can apply to a workflow, which is a set of jobs; to one execution of a workflow; or to a complete pipeline.
The extract task is the same box on the canvas for both ERPs. The way it reads differs, and the table sets the two side by side.
| SAP ECC and S/4HANA | Oracle E-Business Suite (EBS) and Oracle ERP | |
|---|---|---|
| How the extract task reads | Through released function modules, BAPIs and CDS views. A BAPI is a function SAP publishes for reading or writing one business object. A CDS view is a database view SAP defines and releases. | Through JDBC, the standard database connection, with a large table split into partitions so that several threads read it at once. |
| How changes reach it | Application-level change capture: change pointers, table-level change capture, delta-enabled CDS views and extractor delta queues. On the showcase estate this delivers 2.8M events a day. | A last-update column on each table, or LogMiner change capture, which reads changes from the database log where the database administrator allows it. On the showcase estate LogMiner delivers 0.9M events a day. |
| What starts it | A schedule for the nightly delta, or an event when change capture delivers changes during the day. | A schedule for the nightly read past the last-update marker, or an event when LogMiner delivers changes. |
Each job is scheduled against the business event it serves. Backbone works backwards from that event and watches the run against it.
A business deadline is the time of the business event a job serves: the planning dataset must be loaded before the 06:00 MRP run, or the OTIF figures must be on the board by 06:30. Backbone schedules each job against that time and watches the run against it. The five steps on the right show how a deadline is set and kept.
OTIF stands for on time and in full, a delivery measure. MRP is the ERP’s material requirements planning run. Both are business events with a fixed time, and both appear in the table of deadlines below.
| The business event | Run window | What can pause the job | Who is told, and when |
|---|---|---|---|
| The 06:00 MRP runillustration | Opens after the day’s postings; must close before 06:00; a checkpoint after every step | A quality rule on order quantities; the source system unavailable | The on-call planner through PagerDuty when the projected miss is over 15 minutes |
| OTIF on the board by 06:30showcase estate | 00:00 to 06:30; three dependent pipelines | Negative quantities; a changed carrier code | Slack #data-ops when the rule fails; PagerDuty when a miss is projected |
| The finance net-revenue view by 04:00showcase estate | 02:00 to 04:00 every night | A control-total mismatch; a certified view back in Draft | The finance lead and data engineering; ServiceNow records the status change |
| The 3PL tracking feedshowcase estate | Every 15 minutes, with 15 minutes of slack | The partner API unavailable | Slack #data-ops when the lag passes the limit |
One night on the showcase estate, minute by minute.
WMS.DISPATCH_RECORDS.FULFILLED_QTY, the quantity sent per dispatch. A dispatched quantity below zero fails the rule, and 847 rows in one batch mark the whole batch as wrong.otif_nightly, inventory_refresh and 3pl_enrichment all read the warehouse feed. Each one stops at its last checkpoint and keeps its place. The 847 rows become exception records with the rule and the batch attached.otif_nightly completesThe certified view v_otif_daily is refreshed and the supply-chain report is produced. The 84 subscribers of the view receive the usual result.A paused job keeps its place and its checkpoint. A checkpoint is the saved position of the last completed step. When the data is fixed, the job continues from there, so the 841K events that arrived during the hold were applied once, in order.
The hold reached only the pipelines that read the warehouse feed. The finance pipeline that night read other tables and finished on time at 03:48.
The 847 rows kept their full context as exception records, with the rule that failed and the batch they came from. What happens to a row that fails a rule →
The same hold applies to any source. A batch from SAP, from Oracle, from a file or from a webhook that fails a quality rule holds the pipelines that depend on it, and they resume from their checkpoints once it is fixed.
Analysts draw the flow as boxes and arrows. Engineers review it. It runs as an Airflow DAG.
A flow is drawn on the canvas as boxes and arrows. Each box is a task, and each arrow says which task runs after which. The drawing runs as a DAG in Apache Airflow. A DAG is a flow chart of tasks and the order they run in; the letters stand for directed acyclic graph, which means the arrows point one way and the chart has no loops.
Business users build the flow on the canvas with readable labels, such as every weekday at midnight. Engineers review it, add Python where it is needed, and keep the tests. On the showcase estate a supply-chain analyst built the OTIF flow, and data engineering checked it with five golden tests over 218K rows.
otif_nightly flow on the showcase estate as it ran on the night of the hold. SAP changes and the warehouse ingest feed a quality check, then the OTIF calculation, the certified view, and the three outputs that read it. The dashed line is the 06:30 deadline.You keep Airflow. Lima tasks run inside your DAGs, beside the tasks you already have.
Backbone runs on Apache Airflow, the open-source orchestrator most data teams already use. Lima pipelines appear as tasks inside your DAGs. One DAG can run an SAP extract, an Oracle extract, a quality check, a dbt transformation and a BI refresh in sequence, under one retry policy and on one run dashboard. dbt is a tool modelling teams use to write transforms in SQL; a BI refresh rebuilds the reporting dashboards.
An enterprise scheduler can stay in charge of the company calendar. It starts Lima jobs through the API, the interface a system publishes for other programs to call, or through a native task, and it receives the status and the completion event back through webhooks.
Backbone runs on the Airflow 3 line. The dated support statement is on the Standards page.
One dashboard shows every pipeline: its state, its deadline, its latest run, and what it is held on.
An alert carries its context: the rule that failed, the rows that failed it, the upstream step they came from, and the pipelines held as a result. Alerts reach Slack, Teams, PagerDuty and ServiceNow as structured cards.
Root cause uses lineage. Lineage is the record of which tables each table was built from, captured as the pipeline runs. When a rule fails, the alert lists the downstream subscribers of the affected data. On the night of the hold that list held 84 subscribers.
The dashboard on the right shows every pipeline with its trigger, its deadline, its latest run and its state. A held pipeline shows what it is held on, and its clock records the hold.
| Pipeline | Starts | Deadline | Latest run | State |
|---|---|---|---|---|
| otif_nightly | 00:00 daily | 06:30 | 03:14 | On time · resumed from checkpoint |
| sap_cdc_stream | on event | under 2 min | 1m 48s | Streaming |
| inventory_refresh | every 4 hours | soft | held | Held · hold recorded on the clock |
| finance_net_revenue | 02:00 daily | 04:00 | 03:48 | On time · 12 min early |
| 3pl_tracking_sync | every 15 min | +15 min | 14m 22s | On time |
| Signal | Sent when | Delivered to | Showcase estate |
|---|---|---|---|
| quality.anomaly | A quality rule fails on a supply-chain view | Slack #data-ops, as a card with the rule, the fields and the number of downstream subscribers | 340 ms after the rule fails |
| sla.risk | A run is projected to miss its deadline by more than 15 minutes | PagerDuty, with the pipeline, the delay and the deadline | projected miss > 15 min |
| anomaly.resolved | The hold clears and the pipelines resume | Slack #data-ops | at resume |
| cert.status | A certified view changes status | ServiceNow, as a record | on each change |
| product.publish | A data product is published | Microsoft Teams | on each publish |
You keep it. Lima tasks run inside your DAGs with your operators, your retry policies and your deployment. Backbone adds deadlines on the business calendar, pause and resume on a data problem, checkpoint resume with ordered replay, and exception records with owners.
Backbone projects the finish time from live progress while the job runs. When the projected finish passes the deadline by more than 15 minutes, the owner is paged through PagerDuty with the pipeline, the delay and the deadline. The alert arrives before the miss, so the owner can act while there is still time.
The job restarts itself. When the batch that held it passes the rule again, the hold clears and the job continues from its checkpoint. On the showcase estate the three pipelines resumed at 02:58 with 0 manual restarts. A person can also pause or release a job by hand from the run dashboard.
Yes. The scheduler starts a Lima job through the API or a native task and receives the status and the completion event back through webhooks. The company calendar stays where it is, and Backbone keeps the deadline, the checkpoints and the replay for the jobs it runs.
A hold stops the pipelines that depend on the held data. Pipelines that read other sources keep running. On the night of the hold, the three pipelines that read the warehouse feed paused, and the finance pipeline, which reads other tables, finished on time at 03:48.
Thirty minutes on your own estate: one pipeline, one deadline, one quality rule. We show the hold, the resume from the checkpoint, and the replay.