Atlas · GenAI 2026
Data Engineering & Pipelines
41 skills · ontology graph below shows relations within this section.
What this domain covers
This edition groups 41 capabilities in Data Engineering & Pipelines across 19 named categories. The inventory contains 23 concepts and 18 tools; 17 skills appeared in at least two of the three original research runs. The remaining entries stay visible with their lower coverage so a reader can distinguish taxonomy scope from research-system agreement.
Current category labels: Data Architecture · Data Governance & Catalog · Data Governance & Contracts · Data Ingestion · Data Modeling & Design · Data Privacy & Compliance · Data Quality & Contracts · Data Quality & Integration · and 11 more
Frequent learning foundations
- SQL supports 4 mapped skills
- Data Curation supports 3 mapped skills
- Python supports 3 mapped skills
- Apache Kafka supports 1 mapped skill
- Dataset Engineering supports 1 mapped skill
Skills in this section
Data Mesh
Data Architecture
Databricks Unity Catalog
Data Governance & Catalog
Data Contracts
Data Governance & Contracts
Document Parsing
Data Ingestion
Web Scraping
Data Ingestion
Data Modeling
Data Modeling & Design
PII Management
Data Privacy & Compliance
Data Quality Management
Data Quality & Contracts
Data Observability
Data Quality & Integration
Entity Resolution
Data Quality & Integration
NoSQL
Databases & Storage
Data Curation
Dataset Curation
Data Labeling & Annotation
Dataset Curation
Dataset Engineering
Dataset Curation
Evaluation Data Engineering
Dataset Curation
Training Data Curation
Dataset Curation
Apache Spark
Distributed Processing
ETL Pipeline Design
ETL/ELT
Apache Kafka
Streaming
Stream Processing
Streaming
Event-Driven Architecture
Streaming & Messaging
Apache Iceberg
Table Formats & Storage
dbt
Transformation
Data Versioning
Versioning & Lineage
BigQuery
Warehouses & Lakehouses
Databricks
Warehouses & Lakehouses
Snowflake
Warehouses & Lakehouses
Apache Airflow
Workflow Orchestration
BeautifulSoup
Data Ingestion
Data Ingestion
Data Ingestion
Optical Character Recognition (OCR)
Data Ingestion
Scrapy
Data Ingestion
Tesseract
Data Ingestion
CVAT
Dataset Curation
Data Augmentation
Dataset Curation
Label Studio
Dataset Curation
Kedro
ETL/ELT
DVC
Versioning & Lineage
Dagster
Workflow Orchestration
Prefect
Workflow Orchestration
Workflow Orchestration
Workflow Orchestration