Building and running pipelines that collect, transform, store, and serve data as reliable infrastructure.
Data Engineering is the competency of collecting data from diverse sources, designing and implementing ETL/ELT pipelines, and building and operating data warehouses and data lakes. It encompasses schema design, data quality management, workflow orchestration, and real-time streaming processing, with the goal of providing reliable data infrastructure so that analysts and data scientists can access trustworthy data in a timely manner.
Defines Data Engineering from Level 2 (Assist) to Level 6 (Initiate, influence), specifying pipeline design, implementation, and strategic responsibility scope at each level.
Details technical requirements, responsibility scope, and autonomy levels across Junior, Intermediate, Senior, Staff, and Principal stages for L1-L7 mapping.
Validates mid-to-senior engineer competency across 5 domains: data processing system design, ingestion/processing, storage, analysis readiness, and workload automation.
Defines 11 data management knowledge areas (governance, quality, metadata, etc.), providing authoritative grounding for L5-L6 governance/strategy checklists and L4 schema/quality management items.
Systematic mapping of 25 papers classifying data engineering lifecycle activities (collection, transformation, storage, serving) with technical solutions and architectures, grounding L3-L5 checklist behaviors.
