Data Engineering

This is the entry point into data engineering, where you explore foundational concepts. You learn basic SQL syntax, understand pipeline concepts, and grasp the ETL (Extract, Transform, Load) workflow. You can identify relational database table structures and perform simple data extraction and transformation tasks with guidance.

• Can query data from 2 or more tables using SQL SELECT, WHERE, JOIN, and GROUP BY statements
• Can explain the three stages of ETL: Extract, Transform, and Load with concrete examples
• Can diagram primary key and foreign key relationships in a relational database
• Can explain the differences between common data formats (CSV, JSON) and select the appropriate one for a given use case
• Can execute scripts with guidance to move small datasets from one source to another storage system
• Can explain the differences between a data warehouse and a data lake

If you've checked off most of this list, you can implement batch processing ETL pipelines following guidance. You apply data cleansing rules to handle missing values and duplicates, and load structured data into a data warehouse. You use version control tools (e.g., Git) to manage pipeline code and can write basic data quality validation queries.

• Can build at least one batch ETL pipeline using Python or SQL
• Can write cleansing logic to detect and handle missing values, duplicates, and format errors
• Can create tables in a data warehouse using star schema or snowflake schema design
• Can use version control tools (e.g., Git) to manage pipeline code with branches and track change history
• Can write validation queries after loading to verify row counts, null ratios, and unique key integrity
• Can review pipeline execution logs to identify failure causes and re-run pipelines

If you've checked off most of this list, you use workflow orchestration tools (e.g., Airflow, Dagster) to implement pipeline scheduling, dependency management, and retry logic. You apply dimensional modeling techniques to design analytical data models and manage unstructured data in data lakes. You configure monitoring dashboards to track pipeline health and independently resolve routine operational issues.

• Can author and schedule 3 or more DAGs using an orchestration tool (e.g., Airflow, Dagster)
• Can design analytical data models by distinguishing between fact tables and dimension tables
• Can store and query data in a data lake using columnar file formats such as Parquet or ORC
• Can set up failure alerts and implement automatic retry logic for pipelines
• Can explain the difference between Slowly Changing Dimension (SCD) Type 1 and Type 2 and apply the appropriate approach
• Can document data lineage to trace the flow from source to final table
• Can configure monitoring dashboards to visualize pipeline success rate, processing time, and row count trends

If you've checked off most of this list, you go beyond batch processing to design and implement real-time streaming pipelines (e.g., Kafka, Flink). You establish schema evolution strategies that maintain backward compatibility while modifying data structures. You automate pipeline testing and deployment through CI/CD, and lead cost optimization in cloud environments. This corresponds to the Google Cloud Professional Data Engineer certification level.

• Can build a real-time streaming pipeline using a message broker (e.g., Kafka, Kinesis)
• Can operate a schema registry and apply compatibility rules (backward, forward)
• Can include unit tests and integration tests for data pipelines in CI/CD for automated execution
• Can apply partitioning and clustering strategies to reduce query costs by 30% or more
• Can analyze the trade-offs between Lambda and Kappa architectures combining batch and streaming, and select the appropriate approach
• Can define data contracts to specify producer-consumer interfaces
• Can monitor cloud resource usage and produce monthly cost reports with optimization recommendations

If you've checked off most of this list, you design the organization's data platform architecture and lead the adoption of distributed architecture patterns such as data mesh or data fabric. You establish data governance policies to systematize quality, security, and access control, and operate a data catalog. You mentor junior and mid-level engineers and guide the team's technical decision-making.

• Can analyze the feasibility of data mesh or data fabric architecture and develop an adoption plan
• Can document data governance policies (ownership, quality standards, access control, retention periods) and enforce them across the team
• Can build a data catalog that makes organizational data assets searchable and manageable
• Can mentor 3 or more junior/mid-level data engineers and conduct code reviews
• Can define data platform SLAs (Service Level Agreements) and maintain 99.5% or higher availability
• Can assess technical debt, prioritize remediation, and create a quarterly improvement roadmap
• Can collaborate with cross-functional teams (analytics, ML, product) to gather data requirements and incorporate them into the platform
• Can intuitively classify the root cause category of data quality issues (source change, schema drift, infrastructure failure) from pipeline monitoring alert patterns alone

If you've checked off most of this list, you formulate data strategies that influence entire industries beyond a single organization. You demonstrate thought leadership by presenting at data engineering conferences or publishing technical content. You contribute to the evolution of industry-standard frameworks, lead large-scale data infrastructure modernization efforts, and drive regulatory compliance strategies.

• Can develop a 3-year data strategy roadmap for the organization and gain executive approval
• Can present at data engineering conferences or publish 2 or more technical papers
• Can incorporate regulatory requirements (GDPR, data protection laws) into pipeline design and establish audit response frameworks
• Can lead a data infrastructure modernization (legacy migration) project for an organization of 100+ people
• Can submit feedback or contributions to industry-standard frameworks (SFIA, DAMA DMBOK, etc.)
• Can proactively analyze the impact of schema changes on downstream consumers and establish a process that guarantees compatibility through data contracts
• Can quantitatively analyze the cost-performance tradeoff of data pipelines and propose designs that simultaneously achieve cost optimization and SLA compliance

If you've checked off most of this list, you make historic contributions to the field of data engineering. You create widely adopted open-source frameworks or propose new data processing paradigms (like MapReduce or data mesh) that redirect the entire industry. You serve as a bridge between academic research and industry practice, and build educational systems that cultivate the next generation of data engineers.

• Can create or make core contributions to an open-source data framework or tool adopted by 1,000+ organizations
• Can propose a new architectural pattern or methodology for data processing that is cited and adopted across the industry
• Can author a textbook or reference guide for data engineering that is adopted by educational institutions
• Can serve in a technical direction-setting role at an international standards body or major foundation (Apache, Linux Foundation, etc.)
• Can design and operate an educational program that has directly or indirectly trained 100+ data engineers
• Can design and operate organization-level data mesh governance (domain ownership, self-serve platform, data product quality standards)