Atlas · GenAI 2026

ETL Pipeline Design

ETL/ELT pipeline design for LLM apps

conceptPeak: 2024ETL/ELTAI consensus: 2/3

Prerequisites

hardSQL
ETL pipelines query, transform, and load data — SQL is the primary language for the T and L steps
hardPython
Pipeline orchestration, custom transformations, and LLM integration are done in Python

Reis, Housley (2022) Fundamentals of Data Engineering — O'Reilly; best overview of modern ETL/ELT patterns

Anthropic Opus

Reis & Housley (2022) to najlepsza ksiazka. W GenAI: ingest pipelines dla RAG, incremental updates

OpenAI Deep Research

Powtarzalne procesy ingest (incremental) [OA#39]

← is an instance of: dbt(2/3)
→ is subcategory of: Data Engineering(1/3)
← is an instance of: Apache Airflow(1/3)
← is subcategory of: Workflow Orchestration(0/3)
← is subcategory of: Data Ingestion(0/3)
← is an instance of: Google Cloud Data Fusion(0/3)
← is an instance of: Kedro(0/3)