Snowflake is a cloud data warehouse. Lima writes Parquet files to an internal stage, loads them with COPY INTO into RAW tables, merges into CONFORMED tables and keeps a HISTORY schema for full-history migrations. Lima also reads Snowflake through JDBC when the warehouse is the source for a planning or reporting load.
Each method has a job. The first load uses the bulk method; the daily run uses the incremental or change-capture method; write-back uses the interface the system itself uses.
| Method | What it does | When it is used |
|---|---|---|
| Stage + COPY INTO | Bulk loads Parquet files from an internal stage. | First loads, migrations, nightly copies. |
| MERGE | Applies inserts, updates and deletes by key. | Keeping tables current. |
| JDBC reads | Reads tables and views in partitions. | Snowflake as a source. |
| Snowpipe (optional) | Continuous loading of small files. | Streaming-style landing. |
The objects are the tables, views or records Lima reaches through the methods above. The patterns say how a copy is made and kept current.
| Object | Plain meaning |
|---|---|
| RAW schema | tables as they arrived, typed |
| CONFORMED schema | current state by key |
| HISTORY schema | every version, for archives and audits |
| Pattern | Supported |
|---|---|
| Full load | Yes |
| Incremental | Yes |
| Change capture | applied by MERGE from the source's change feed |
| Streaming | micro-batch landing via Snowpipe |
| Write-back | — |
Getting SAP and Oracle types right on landing: text dates and times, currency decimals, zero-padded keys and client or organisation columns.
The prerequisites are what the team that runs Snowflake grants. The four steps are the same for every connector.
Incoming data = valid records + exception records. A row that fails a check is kept, routed to the person who owns it with a deadline, and replayed once fixed. The load carries on with the valid rows and the report says exactly what was held back. How exceptions work →
Each pair page covers one path end to end: objects out, model in, checks, cadence and the offer.
Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Lima connects from outside through the interfaces listed above with credentials your administrators grant. Where a vendor requires its own agent or add-on for a method, the prerequisites say so.
Parallel reads are capped and the cap is agreed with the team that runs the system. Large reads run inside an agreed run window. Change capture keeps the daily volume small.
A report per run: rows in, rows out, control totals and variance per table, the exceptions held back and who owns them, and whether the deadline was met.
Yes. Most estates start with one source, one target and one dataset, then add tables and patterns on the same connection.
A trial connects this system, runs a first load and ends with a reconciliation report. Your administrators see the prerequisites before anything is installed.
