Amazon S3 is the landing zone for most AWS estates. Lima writes Parquet files partitioned by load date and business keys, registers them in the Glue Data Catalog and, where wanted, writes Apache Iceberg tables so Athena, Redshift Spectrum and Trino read them with snapshots. S3 is also read as a source for files other systems drop there.
Each method has a job. The first load uses the bulk method; the daily run uses the incremental or change-capture method; write-back uses the interface the system itself uses.
| Method | What it does | When it is used |
|---|---|---|
| Partitioned Parquet writes | Writes files by partition in parallel. | Archives, migrations, landing. |
| Iceberg tables | Writes an Iceberg table with snapshots. | Queryable archives. |
| Glue catalog registration | Registers tables and partitions. | Every load. |
| File reads | Reads files other systems drop. | S3 as a source. |
The objects are the tables, views or records Lima reaches through the methods above. The patterns say how a copy is made and kept current.
| Object | Plain meaning |
|---|---|
| raw/ prefix | as arrived |
| history/ prefix | every version |
| Iceberg tables | snapshots and schema evolution |
| Pattern | Supported |
|---|---|
| Full load | Yes |
| Incremental | Yes |
| Change capture | appended change files |
| Streaming | micro-batch file writes |
| Write-back | — |
Choosing partitions that suit both the archive and the queries that will read it.
The prerequisites are what the team that runs Amazon S3 grants. The four steps are the same for every connector.
Incoming data = valid records + exception records. A row that fails a check is kept, routed to the person who owns it with a deadline, and replayed once fixed. The load carries on with the valid rows and the report says exactly what was held back. How exceptions work →
Each pair page covers one path end to end: objects out, model in, checks, cadence and the offer.
Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Landing and history: full history landed with a count check per table, then kept current.
Open the pair page →Lima connects from outside through the interfaces listed above with credentials your administrators grant. Where a vendor requires its own agent or add-on for a method, the prerequisites say so.
Parallel reads are capped and the cap is agreed with the team that runs the system. Large reads run inside an agreed run window. Change capture keeps the daily volume small.
A report per run: rows in, rows out, control totals and variance per table, the exceptions held back and who owns them, and whether the deadline was met.
Yes. Most estates start with one source, one target and one dataset, then add tables and patterns on the same connection.
A trial connects this system, runs a first load and ends with a reconciliation report. Your administrators see the prerequisites before anything is installed.
