Build and maintain data pipelines (ETL/ELT)
Ingest data from multiple sources (APIs, relational databases, logs, files, events)
Transform and standardize data (cleaning, deduplication, enrichment, business rules)
Load data into analytical destinations (Data Warehouse / Lakehouse / Data Lake)
Orchestrate workflows (scheduling, dependencies, reprocessing/backfill)
Data modeling and provisioning for analysis
Dimensional modeling (star schema) and/or normalized models as appropriate
Create layers (raw/bronze, refined/silver, curated/gold)
Define tables, partitioning, indexes and performance/cost strategies
Publish “ready-to-consume” datasets for BI, product teams and data science
Ensure quality, reliability and observability
Data testing (schema validation, null checks, range checks, referential integrity)
Pipeline monitoring (SLAs, latency, volume, failures, delayed data)
Lineage (data lineage) and data catalog/documentation
Incident handling and root cause analysis
Cross-functional collaboration
Work with data analysts, data scientists, software engineers and stakeholders. Translate business needs into data contracts (definitions, metrics, SLAs).

Strong business domain understanding (ability to interpret metrics and impact)
Ability to reduce cost and improve performance (query and storage optimization)
Observability best practices (monitoring, useful alerts, clear SLAs)
Good communication and documentation (well-described, reusable datasets)
Advanced SQL: complex joins, window functions, CTEs, optimization, modeling
Python (or Scala/Java): processing, integrations, automation, testing
Data structures, knowledge of distributed systems and performance
Relational databases: PostgreSQL / SQL Server, Athena, Redshift
Data Warehouses / Lakehouses (concepts and usage): partitioning, clustering, cost management
Data formats: Parquet / Avro / ORC, compression, schema evolution
Orchestration tools: Airflow, Dagster, Prefect (or equivalents)
Transformation / analytics engineering: dbt (very common)
Distributed processing: Spark (PySpark), and alternatives depending on the stack
Kafka / Kinesis / Pub/Sub, event stream processing
CDC (Change Data Capture) and near-real-time pipeline design
Cloud concepts: networking, IAM, storage, compute
Infrastructure as Code: Terraform
Containers and deployment: Docker (and sometimes Kubernetes)
CI/CD for pipelines and models (tests, linting, controlled deployments)
Testing and validation: Great Expectations, dbt tests, custom checks
Security best practices: least privilege, segregation of duties, auditing

Data Engineer

Key skills