Source: Levelica Canonical: https://www.levelica.com/en/guides/data-engineering # Data Engineering ## Summary Building and running pipelines that collect, transform, store, and serve data as reliable infrastructure. ## Description Data Engineering is the competency of collecting data from diverse sources, designing and implementing ETL/ELT pipelines, and building and operating data warehouses and data lakes. It encompasses schema design, data quality management, workflow orchestration, and real-time streaming processing, with the goal of providing reliable data infrastructure so that analysts and data scientists can access trustworthy data in a timely manner. --- ## Level Overview | Level | Proficiency Stage | Hashtags | | ----- | ---------------------- | ------------------------------------------------------------- | | Lv.1 | Assist, Explore | #SQL #DataPipeline #ETLBasics #SchemaDesign | | Lv.2 | Assist, Apply | #BatchProcessing #DataWarehouse #VersionControl #DataCleaning | | Lv.3 | Apply, Build | #Orchestration #DataModeling #DataLake #Monitoring | | Lv.4 | Advise, Architect | #StreamProcessing #SchemaEvolution #CostOptimization #CI-CD | | Lv.5 | Initiate, Strategize | #DataMesh #Governance #TeamMentoring #PlatformDesign | | Lv.6 | Influence, Standardize | #DAMA-DMBOK #DataStrategy #ComplianceArchitecture #GDPR | | Lv.7 | Set Strategy, Pioneer | #ApacheFoundation #DataParadigm #OpenSourceFramework | > Proficiency stages are mapped from SFIA 8's Assist, Apply, Advise, Initiate/Influence, and Set Strategy responsibility levels, cross-referenced with the UK DDaT Data Engineer role framework, to the Levelica 7-level system. > **Integrated Framework:** This guide's 7-level structure is not a mapping of a single rating system but the integration of four independent axes. SFIA 9 data engineering competency provides the **pipeline maturity axis** (data pipeline complexity, from batch ETL to real-time streaming). The DAMA-DMBOK data management framework provides the **data quality axis** (scope of quality governance, from basic validation to data contracts and data mesh). Kleppmann's data systems theory provides the **system reliability axis** (scalability and fault tolerance of data infrastructure, from single database to distributed systems). And practitioner expertise provides the **organizational data culture axis** (operational intuition not captured by frameworks, including pipeline failure instinct, schema evolution impact prediction, and cost-performance tradeoffs). Each level boundary is determined at the intersection of these four axes. --- ## Level 1 #SQL #DataPipeline #ETLBasics #SchemaDesign ### Summary Learning SQL and ETL basics while exploring how data flows. ### Description This is the entry point into data engineering, where you explore foundational concepts. You learn basic SQL syntax, understand pipeline concepts, and grasp the ETL (Extract, Transform, Load) workflow. You can identify relational database table structures and perform simple data extraction and transformation tasks with guidance. ### Checklist - [ ] Can query data from 2 or more tables using SQL SELECT, WHERE, JOIN, and GROUP BY statements - [ ] Can explain the three stages of ETL: Extract, Transform, and Load with concrete examples - [ ] Can diagram primary key and foreign key relationships in a relational database - [ ] Can explain the differences between common data formats (CSV, JSON) and select the appropriate one for a given use case - [ ] Can execute scripts with guidance to move small datasets from one source to another storage system - [ ] Can explain the differences between a data warehouse and a data lake > If you've checked off most of this list, you're ready for the Assist, Apply stage, building batch processing pipelines independently and validating data quality. Bandura(1977)'s Social Learning theory suggests watching experienced data engineers build pipelines and studying ETL case studies builds the confidence to tackle these tasks yourself. > **Cross-Validation:** Three independent reference frameworks converge on the same level boundaries. (1) SFIA 8's Assist→Apply transition and UK DDaT's Data Engineer→Senior transition align at the L2↔L3 boundary (batch pipeline building→orchestration automation). (2) GitLab Career Ladder's Senior→Staff transition and Databricks Associate→Professional certification boundary converge at the L4↔L5 boundary (streaming design→platform strategy). (3) SFIA's Influence level and DAMA-DMBOK's strategic data management stage align at the L6↔L7 boundary (industry standard contribution→paradigm pioneering). --- ## Level 2 #BatchProcessing #DataWarehouse #VersionControl #DataCleaning ### Summary Building batch pipelines with guidance and validating data quality. ### Description If you've checked off most of this list, you can implement batch processing ETL pipelines following guidance. You apply data cleansing rules to handle missing values and duplicates, and load structured data into a data warehouse. You use version control tools (e.g., Git) to manage pipeline code and can write basic data quality validation queries. ### Checklist - [ ] Can build at least one batch ETL pipeline using Python or SQL - [ ] Can write cleansing logic to detect and handle missing values, duplicates, and format errors - [ ] Can create tables in a data warehouse using star schema or snowflake schema design - [ ] Can use version control tools (e.g., Git) to manage pipeline code with branches and track change history - [ ] Can write validation queries after loading to verify row counts, null ratios, and unique key integrity - [ ] Can review pipeline execution logs to identify failure causes and re-run pipelines > If you've checked off most of this list, you're ready for the Apply, Build stage, using orchestration tools to schedule multiple pipelines and independently designing data models. Kolb(1984)'s Experiential Learning theory suggests repeating the cycle of building pipelines, observing execution results, reflecting, and applying improved designs builds deeper engineering skill. --- ## Level 3 #Orchestration #DataModeling #DataLake #Monitoring ### Summary Orchestrating pipelines and designing data models independently. ### Description If you've checked off most of this list, you use workflow orchestration tools (e.g., Airflow, Dagster) to implement pipeline scheduling, dependency management, and retry logic. You apply dimensional modeling techniques to design analytical data models and manage unstructured data in data lakes. You configure monitoring dashboards to track pipeline health and independently resolve routine operational issues. ### Checklist - [ ] Can author and schedule 3 or more DAGs using an orchestration tool (e.g., Airflow, Dagster) - [ ] Can design analytical data models by distinguishing between fact tables and dimension tables - [ ] Can store and query data in a data lake using columnar file formats such as Parquet or ORC - [ ] Can set up failure alerts and implement automatic retry logic for pipelines - [ ] Can explain the difference between Slowly Changing Dimension (SCD) Type 1 and Type 2 and apply the appropriate approach - [ ] Can document data lineage to trace the flow from source to final table - [ ] Can configure monitoring dashboards to visualize pipeline success rate, processing time, and row count trends > If you've checked off most of this list, you're ready for the Advise, Architect stage, designing real-time streaming pipelines, managing schema evolution, and optimizing costs across data infrastructure. Vygotsky(1978)'s Zone of Proximal Development suggests tackling data modeling or orchestration tasks just beyond your current ability, with senior engineer review, builds skill fastest. --- ## Level 4 #StreamProcessing #SchemaEvolution #CostOptimization #CI-CD ### Summary Designing streaming pipelines and managing schema evolution. ### Description If you've checked off most of this list, you go beyond batch processing to design and implement real-time streaming pipelines (e.g., Kafka, Flink). You establish schema evolution strategies that maintain backward compatibility while modifying data structures. You automate pipeline testing and deployment through CI/CD, and lead cost optimization in cloud environments. This corresponds to the Google Cloud Professional Data Engineer certification level. ### Checklist - [ ] Can build a real-time streaming pipeline using a message broker (e.g., Kafka, Kinesis) - [ ] Can operate a schema registry and apply compatibility rules (backward, forward) - [ ] Can include unit tests and integration tests for data pipelines in CI/CD for automated execution - [ ] Can apply partitioning and clustering strategies to reduce query costs by 30% or more - [ ] Can analyze the trade-offs between Lambda and Kappa architectures combining batch and streaming, and select the appropriate approach - [ ] Can define data contracts to specify producer-consumer interfaces - [ ] Can monitor cloud resource usage and produce monthly cost reports with optimization recommendations > If you've checked off most of this list, you're ready for the Initiate, Strategize stage, designing team-level data platforms, establishing data governance frameworks, and mentoring junior engineers. Ericsson(1993)'s Deliberate Practice theory suggests training specifically on your weak areas like streaming pipeline design, schema evolution management, and cost optimization. --- ## Level 5 #DataMesh #Governance #TeamMentoring #PlatformDesign ### Summary Leading data platform design, governance, and team mentoring. ### Description If you've checked off most of this list, you design the organization's data platform architecture and lead the adoption of distributed architecture patterns such as data mesh or data fabric. You establish data governance policies to systematize quality, security, and access control, and operate a data catalog. You mentor junior and mid-level engineers and guide the team's technical decision-making. ### Checklist - [ ] Can analyze the feasibility of data mesh or data fabric architecture and develop an adoption plan - [ ] Can document data governance policies (ownership, quality standards, access control, retention periods) and enforce them across the team - [ ] Can build a data catalog that makes organizational data assets searchable and manageable - [ ] Can mentor 3 or more junior/mid-level data engineers and conduct code reviews - [ ] Can define data platform SLAs (Service Level Agreements) and maintain 99.5% or higher availability - [ ] Can assess technical debt, prioritize remediation, and create a quarterly improvement roadmap - [ ] Can collaborate with cross-functional teams (analytics, ML, product) to gather data requirements and incorporate them into the platform - [ ] Can intuitively classify the root cause category of data quality issues (source change, schema drift, infrastructure failure) from pipeline monitoring alert patterns alone > If you've checked off most of this list, you're ready for the Influence, Standardize stage, establishing industry-level data strategies, contributing to industry standards, and exerting influence across organizations. Schön(1983)'s Reflective Practice theory suggests checking and revising your design assumptions in real time as you build data platforms and establish governance. --- ## Level 6 #DAMA-DMBOK #DataStrategy #ComplianceArchitecture #GDPR ### Summary Shaping industry-level data strategy and contributing to standards. ### Description If you've checked off most of this list, you formulate data strategies that influence entire industries beyond a single organization. You demonstrate thought leadership by presenting at data engineering conferences or publishing technical content. You contribute to the evolution of industry-standard frameworks, lead large-scale data infrastructure modernization efforts, and drive regulatory compliance strategies. ### Checklist - [ ] Can develop a 3-year data strategy roadmap for the organization and gain executive approval - [ ] Can present at data engineering conferences or publish 2 or more technical papers - [ ] Can incorporate regulatory requirements (GDPR, data protection laws) into pipeline design and establish audit response frameworks - [ ] Can lead a data infrastructure modernization (legacy migration) project for an organization of 100+ people - [ ] Can submit feedback or contributions to industry-standard frameworks (SFIA, DAMA DMBOK, etc.) - [ ] Can proactively analyze the impact of schema changes on downstream consumers and establish a process that guarantees compatibility through data contracts - [ ] Can quantitatively analyze the cost-performance tradeoff of data pipelines and propose designs that simultaneously achieve cost optimization and SLA compliance > If you've checked off most of this list, you're ready for the Set Strategy, Pioneer stage, pioneering new paradigms in data engineering and leaving a legacy that impacts future generations. Argyris & Schön(1978)'s Double-Loop Learning theory suggests questioning the assumptions behind data strategy and compliance architecture to create stronger industry standards. --- ## Level 7 #ApacheFoundation #DataParadigm #OpenSourceFramework ### Summary Pioneering new data paradigms and leaving a lasting legacy. ### Description If you've checked off most of this list, you make historic contributions to the field of data engineering. You create widely adopted open-source frameworks or propose new data processing paradigms (like MapReduce or data mesh) that redirect the entire industry. You serve as a bridge between academic research and industry practice, and build educational systems that cultivate the next generation of data engineers. ### Checklist - [ ] Can create or make core contributions to an open-source data framework or tool adopted by 1,000+ organizations - [ ] Can propose a new architectural pattern or methodology for data processing that is cited and adopted across the industry - [ ] Can author a textbook or reference guide for data engineering that is adopted by educational institutions - [ ] Can serve in a technical direction-setting role at an international standards body or major foundation (Apache, Linux Foundation, etc.) - [ ] Can design and operate an educational program that has directly or indirectly trained 100+ data engineers - [ ] Can design and operate organization-level data mesh governance (domain ownership, self-serve platform, data product quality standards) > If you've checked off most of this list, you're at the forefront of data engineering, shaping how the technology ecosystem processes and manages data at scale. Bloom(1956)'s Taxonomy suggests that designing new data processing paradigms or open-source frameworks represents the highest cognitive level of Create, leaving a legacy that influences future generations. --- ## References - [SFIA 8 — Data Engineering Skill Definition (Levels 2-6)](https://sfia-online.org/en/sfia-8/skills/data-engineering) — SFIA Foundation - [GitLab Data Engineer Career Ladder (Junior → Principal)](https://handbook.gitlab.com/job-families/marketing/enterprise-data/data-engineer/) — GitLab - [Google Cloud Professional Data Engineer Certification](https://cloud.google.com/learn/certification/data-engineer) — Google Cloud - [DAMA-DMBOK: Data Management Body of Knowledge (2nd Edition)](https://dama.org/learning-resources/dama-data-management-body-of-knowledge-dmbok/) — DAMA International - [What About the Data? A Mapping Study on Data Engineering for AI Systems (CAIN 2024)](https://arxiv.org/abs/2402.05156) — IEEE/ACM CAIN Conference